Python

Python
Segmentation fault issue when using Miniconda on ITO-A

After installing Miniconda (Anaconda) on the ITO-A supercomputer and creating a virtual environment, a problem of Segmentation fault occurred when running a Python script. A solution to this problem was found and summarized here. How to revolve Segmentation fault issue The problem was probably the conflict with the vendor installed Python. This issue was resolved by appending the following in last part of ~/.bashrc: export PYTHONPATH= The problem was resolved by logging in again and installing Miniconda as usual. The specific method is described here (in Japanese and please translate using an online tool). Occurrence of segmentation fault The environment used to work fine, but when I tried to work […]

Read more
Python
Exporting and rebuilding conda virtual environment

Background After installing Miniconda on a new Windows 10 PC and setting up a virtual environment for running xhycom, xarray.Dataset.hvPlot does not work. There seems to be a problem with the handling of datetime64; pandas get_loc(key) returns a KeyError in datetimes.py. get_loc(key) seems to take an index and there seems to be a confusion between the index and the integer value of datetime64(ns). Since I do not know the solution, I decide to rebuild the virtual environment that is working well. This problem has been discussed in this site. It may be associated with some incompatibility between xarray 0.16 and pandas 1.1 (works with pandas 1.05). Exporting the virtual environment […]

Read more
Computing
Mounting G Suite Google drive on Windows PC

Background FApps has been introduced in the Graduate School of Frontier Sciences, the University of Tokyo. I was considering how to access data on the shared drive (formerly called team drive) of G Suite from a local Python code on Windows 10 Pro using PyDrive. However, it ended in failure. In addition, uninstalling PyDrive installed with conda also failed. Recently, I learned that there is a way to mount Google shared drive locally with Google Drive File Stream as a way to access the data on the shared drive. Google drive file stream Login with your FApps account and open Google Drive. There are My Drive and Shared Drive. Mounting […]

Read more
Computing
Bar plots on Google Earth Pro

Creating Bar plots kml files for Google Earth Pro using Python googleearthplot referring to this site. Install Suppose pandas is installed. Install simplekml and googleearthplot. pip install simplekml pip install googleearthplot googleearthplot is a Python2 code and some modification is required in Python3 because of an error associated with print statement. First find the location of installed googleearthplot as follows: pip show googleearthplot Edit googleearthplot.py to modify the lines of print statement by converting print **** to print(****) using parentheses. Usage Open a Jupyter Notebook and import packages. Referring to the original site, some examples are shown. Double cricking on created kml file, a plot should be presented on Google […]

Read more
Python
Time series plotting using hvPlot

hvPlot, a component of PyViz ia a new interactive plotting tool, which is a wrapper of Bokeh. hvPlot is easy to customize plotting options. However, currently there are some drawbacks, such as difficulty in adjusting aspect ratio of graph area, which may be resolved in Bokeh 1.1, slow processing, and cumbersome in exporting high resolution PNG file, comparing to Matplotlib. Default time series plotting Suppose df is a pandas DataFrame having a time series index and columns of 'kion' and 'sped', which are actually a dataset of air temperature and wind speed at Chiba Observatory, Japan, provided by Japan Meteorological Agency. import hvplot.pandas df[['kion','sped']].hvplot() Customized plotting Now one column of […]

Read more
Python
Import module or package in Python

Suppose that a main script of [cci]main.py[/cci] (or a corresponding [cci]Jupyter Notebook[/cci]) exists in the directory of [cci]myproject[/cci], and its subdirectory [cci]sub[/cci] contains modules of [cci]mod_a.py[/cci] (a function of [cci]func_a ()[/cci] included) and [cci]mod_b.py[/cci] (a function of [cci]func_b()[/cci] included) where [cci]main.py[/cci] imports the module [cci]mod_a[/cci] and the module [cci]mod_a[/cci] imports the module [cci]mod_b[/cci]. First, creating an empty file [cci]__init__.py[/cci] in the directory [cci]sub[/cci] so that [cci]sub[/cci] is recognized as a package as shown below: [cc] myproject/ main.py sub/ __init__.py mod_a.py (func_a()) mod_b.py (func_b()) [/cc] The sub-directory, [cci]sub[/cci], containing two modules, becomes a package when [cci]__init__.py[/cci] is included. Import [cci]mod_a.py[/cci] in [cci]main.py[/cci] The following is how to import the module [cci]mod_a.py[/cci] […]

Read more
Computing
File output in Bokeh

Updated on September 23, 2020:There seems a bug in bokeh 2.2.1, in which png files do not contain whole of the graphic area when exporting to png from html. To avoid this error, use bokeh 1.4.0, which can be insalled with conda install -c pyviz bokeh==1.4.0 in a new conda virtual environment This article is targeted for Windows 10. Although Matplotlib is a conventional plotting tool in Python, Bokeh is a relatively new package that makes it easy to create interactive plots. Unfortunately, the image output for pasting into PowerPoint or Word is low resolution by default, and it is necessary to enhance the resolution by setting the value of […]

Read more
Computing
Set up ITO-A@kyushu-u supercomputer

This is a memorandum report for seting up an environment on ITO-A at Kyushu University Supercomputer Center. How to login On Windows, you need a terminal emulator to login and MobaXterm is recommended to use, which also has a function of X11 forwarding for displaying X Window screen. Its installation method is introduced in this site (in Japanese). Change file access permission You may change the permission of your directories and files as follows so that group members can read and execute files and help you: chmod -R 750 /home/usr_num/account where the first, second, and third digits are types of permissions for user, group, and all the others, and the […]

Read more