Skip to content

Instantly share code, notes, and snippets.

@gijzelaerr
Created February 20, 2014 13:17
Show Gist options
  • Save gijzelaerr/9113306 to your computer and use it in GitHub Desktop.
Save gijzelaerr/9113306 to your computer and use it in GitHub Desktop.
python batch_test.py
========== Removing files
========== $ rm -fr WSRT.MS* WSRT*img WSRT*fits
========== Running makems
========== $ makems /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT_makems.cfg
Debug: registered context Global=0
Debug::loadLevels: error opening makems.debug
2014-02-20 13:15:59 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657) Leap second table TAI_UTC seems out-of-date.
2014-02-20 13:15:59 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657)+ Until table is updated (see aips++ manager) times and coordinates
2014-02-20 13:15:59 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657)+ derived from UTC could be wrong by 1s or more.
Created 1 MS part and its VDS file
Created global VDS file WSRT.MS.gds
========== $ mv WSRT.MS_p0 WSRT.MS
========== $ ls WSRT.MS
ANTENNA FIELD OBSERVATION PROCESSOR table.dat table.f1 table.f3 table.f4_TSM0 vis.DATA
DATA_DESCRIPTION FLAG_CMD POINTING SPECTRAL_WINDOW table.f0 table.f2 table.f3_TSM0 table.info vis.flg
FEED HISTORY POLARIZATION STATE table.f0i table.f2_TSM0 table.f4 table.lock vis.uvw
========== $ /usr/bin/owlcat.sh downweigh-redundant-baselines WSRT.MS
Successful read/write open of default-locked table WSRT.MS: 22 columns, 7560 rows
Successful readonly open of default-locked table /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS/ANTENNA: 8 columns, 14 rows
00:00:00: Looking for redundant baselines
105 baselines
Baseline 0m, 14 ifrs: 0-0 1-1 2-2 3-3 4-4 5-5 6-6 7-7 8-8 9-9 10-10 11-11 12-12 13-13
Baseline 72m, 2 ifrs: 10-11 12-13
Baseline 144m, 9 ifrs: 0-1 1-2 2-3 3-4 4-5 5-6 6-7 7-8 8-9
Baseline 288m, 8 ifrs: 0-2 1-3 2-4 3-5 4-6 5-7 6-8 7-9
Baseline 432m, 7 ifrs: 0-3 1-4 2-5 3-6 4-7 5-8 6-9
Baseline 576m, 6 ifrs: 0-4 1-5 2-6 3-7 4-8 5-9
Baseline 720m, 5 ifrs: 0-5 1-6 2-7 3-8 4-9
Baseline 864m, 4 ifrs: 0-6 1-7 2-8 3-9
Baseline 1008m, 3 ifrs: 0-7 1-8 2-9
Baseline 1152m, 2 ifrs: 0-8 1-9
Baseline 1296m, 3 ifrs: 0-9 10-12 11-13
Baseline 1332m, 2 ifrs: 0-10 9-12
Baseline 1404m, 2 ifrs: 0-11 9-13
00:00:00: Closing MS.
========== $ lwimager ms=WSRT.MS data=CORRECTED_DATA mode=channel weight=natural npix=10
lwimager: Version 20130816-OMS
2014-02-20 13:15:59 INFO Imager::open() Opening MeasurementSet /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS
2014-02-20 13:15:59 INFO imager::data selection Selecting data
2014-02-20 13:16:00 INFO imager::data selection Selected 6552 out of 7560 visibilities.
2014-02-20 13:16:00 INFO Imager::selectDataChannel() Selecting channel 0 for spw 0
2014-02-20 13:16:00 INFO imager::defineimage() Defining image properties:nx=10 ny=10 cellx='1arcsec' celly='1arcsec' stokes=I' mode=channel nchan=1 start=0 step=1 spwids=[0] fieldid=0 facets=1 frame=1 distance='0m'
2014-02-20 13:16:00 INFO imager::defineimage() phaseCenter='field-0 ' mStart='Radialvelocity: 0' qStep='1 km/s'' mFreqStart='Frequency: 0
2014-02-20 13:16:00 INFO imager::weight() Weighting MS: Imaging weights will be changed
2014-02-20 13:16:00 INFO imager::weight() Natural weighting
2014-02-20 13:16:00 INFO imager::setoptions() Setting processing options
2014-02-20 13:16:00 INFO imager::makeimage() Calculating image (without full skyequation)
2014-02-20 13:16:00 INFO imager::makeimage() Making dirty image from corrected data
2014-02-20 13:16:00 INFO imager::makeimage() Image is : WSRT-I-channel1.img
2014-02-20 13:16:00 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Did not get the position of LOFAR from data repository
2014-02-20 13:16:00 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Please contact CASA to add it to the repository.
2014-02-20 13:16:00 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Frequency conversion will not work
2014-02-20 13:16:00 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657) Leap second table TAI_UTC seems out-of-date.
2014-02-20 13:16:00 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657)+ Until table is updated (see aips++ manager) times and coordinates
2014-02-20 13:16:00 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657)+ derived from UTC could be wrong by 1s or more.
2014-02-20 13:16:00 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Did not get the position of LOFAR from data repository
2014-02-20 13:16:00 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Please contact CASA to add it to the repository.
2014-02-20 13:16:00 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Frequency conversion will not work
2014-02-20 13:16:00 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Did not get the position of LOFAR from data repository
2014-02-20 13:16:00 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Please contact CASA to add it to the repository.
2014-02-20 13:16:00 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Frequency conversion will not work
2014-02-20 13:16:00 INFO imager::createFTMachine() Performing interferometric gridding...
lwimager normally ended
========== $ tigger-convert test-lsm.txt --rename --format "ra_d dec_d i q u v" --center 0.1deg,60.5deg -f
Reading test-lsm.txt (ASCII table)
Model contains 2 sources
Center of field set to 0.1deg,60.5deg
Recomputing the 'r' attribute based on the field center
Renaming sources using the COPART convention
Saving model containing 2 sources to test-lsm.lsm.html (Tigger sky model)
========== $ tigger-convert test-lsm.lsm.html test-lsm1.txt --output-format "name ra_h dec_d i q u v freq0 spi rm tags..." -f
Reading test-lsm.lsm.html (Tigger sky model)
Model contains 2 sources
Saving model containing 2 sources to test-lsm1.txt (ASCII table)
========== $ cut -d " " -f 1-10 test-lsm1.txt >test-lsm1.txt.tmp
========== $ diff test-lsm1.txt.tmp /home/gijs/build/meqtrees-cattery/test/Batchtest/test-lsm1.txt.reference
========== $ tigger-convert test-lsm1.txt --format "name ra_h dec_d i q u v freq0 spi rm tags..." -f
Reading test-lsm1.txt (ASCII table)
Model contains 2 sources
Saving model containing 2 sources to test-lsm1.lsm.html (Tigger sky model)
========== $ /usr/bin/owlcat.sh plot-ms WSRT.MS DATA:I -o data_i.png
===> Attaching to MS WSRT.MS
Successful readonly open of default-locked table WSRT.MS: 22 columns, 7560 rows
===> Using all flags
===> Channel selection is slice(0, None, None)
===> Timeslot selection is slice(0, None, None)
Successful readonly open of default-locked table /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS/ANTENNA: 8 columns, 14 rows
===> Selected all 91 interferometers
Successful readonly open of default-locked table /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS/DATA_DESCRIPTION: 3 columns, 1 rows
===> Using first DATA_DESC_ID (0)
Successful readonly open of default-locked table /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS/POLARIZATION: 4 columns, 1 rows
Successful readonly open of default-locked table /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS/SPECTRAL_WINDOW: 14 columns, 1 rows
===> Processing DATA_DESC_ID 0 (1404 MHz): 64x4 by 7560 rows
===> 0 of 1935360 (0%) visibilities are flagged
===> Making plot DATA mean Stokes I
===> Found data for 91 interferometers
===> Wrote data_i.png
========== $ /usr/bin/owlcat.sh run-imager ms=WSRT.MS name_dirty=tmp
run-imager.sh command line: ms=WSRT.MS name_dirty=tmp
cachesize=512
cellsize=1arcsec
channels=
data=DATA
export_column=0
field=0
fixed=0
flux_scale=2
gain=.1
ifrs=
image2fits=image2fits
img_channels=
lwimager=lwimager
mask=
mode=channel
ms=WSRT.MS
name_dirty=tmp
name_model=model
name_prefix=
name_residual=residual
name_restored=restored
name_suffix=
niter=1000
npix=512
oper=image
padding=1.2
phasecenter=
prefervelocity=False
remove_img=1
robust=0
select=
size=
spwid=0
stokes=I
taper=
threshold=0Jy
uvmax=
uvmin=
weight=natural
wprojplanes=0
Using all interferometers
Making dirty image: lwimager ms=WSRT.MS data=DATA operation=image stokes=I mode=channel weight=natural wprojplanes=0 npix=512 cellsize=1arcsec spwid=0 field=0 padding=1.2 cachesize=512 prefervelocity=False robust=0 image=tmp.img
lwimager: Version 20130816-OMS
2014-02-20 13:16:02 INFO Imager::open() Opening MeasurementSet /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS
2014-02-20 13:16:02 INFO imager::data selection Selecting data
2014-02-20 13:16:02 INFO imager::data selection Selected 6552 out of 7560 visibilities.
2014-02-20 13:16:02 INFO Imager::selectDataChannel() Selecting channel 0 for spw 0
2014-02-20 13:16:02 INFO imager::defineimage() Defining image properties:nx=512 ny=512 cellx='1arcsec' celly='1arcsec' stokes=I' mode=channel nchan=1 start=0 step=1 spwids=[0] fieldid=0 facets=1 frame=1 distance='0m'
2014-02-20 13:16:02 INFO imager::defineimage() phaseCenter='field-0 ' mStart='Radialvelocity: 0' qStep='1 km/s'' mFreqStart='Frequency: 0
2014-02-20 13:16:02 INFO imager::weight() Weighting MS: Imaging weights will be changed
2014-02-20 13:16:02 INFO imager::weight() Natural weighting
2014-02-20 13:16:02 INFO imager::setoptions() Setting processing options
2014-02-20 13:16:02 INFO imager::makeimage() Calculating image (without full skyequation)
2014-02-20 13:16:02 INFO imager::makeimage() Making dirty image from observed data
2014-02-20 13:16:02 INFO imager::makeimage() Image is : tmp.img
2014-02-20 13:16:02 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Did not get the position of LOFAR from data repository
2014-02-20 13:16:02 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Please contact CASA to add it to the repository.
2014-02-20 13:16:02 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Frequency conversion will not work
2014-02-20 13:16:02 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657) Leap second table TAI_UTC seems out-of-date.
2014-02-20 13:16:02 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657)+ Until table is updated (see aips++ manager) times and coordinates
2014-02-20 13:16:02 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657)+ derived from UTC could be wrong by 1s or more.
2014-02-20 13:16:02 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Did not get the position of LOFAR from data repository
2014-02-20 13:16:02 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Please contact CASA to add it to the repository.
2014-02-20 13:16:02 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Frequency conversion will not work
2014-02-20 13:16:02 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Did not get the position of LOFAR from data repository
2014-02-20 13:16:02 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Please contact CASA to add it to the repository.
2014-02-20 13:16:02 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Frequency conversion will not work
2014-02-20 13:16:02 INFO imager::createFTMachine() Performing interferometric gridding...
lwimager normally ended
real 0m0.168s
user 0m0.152s
sys 0m0.136s
Success, all *.img files will be removed after conversion to FITS
/usr/bin/run-imager.sh: line 310: image2fits: command not found
importing meqserver
importing Compile
importing TDLOptions
Starting meqserver
========== Compiling /home/gijs/build/meqtrees-cattery/test/Batchtest/testing-sim.py
(Meow.MSUtils: found /usr/bin/lwimager, can use it for imaging.)
(Meow.MSUtils: found image viewer /usr/bin/tigger)
Successful readonly open of autonoread-locked table WSRT.MS: 22 columns, 7560 rows
Successful readonly open of autonoread-locked table /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS/ANTENNA: 8 columns, 14 rows
Successful readonly open of default-locked table /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS/OBSERVATION: 9 columns, 1 rows
Successful readonly open of autonoread-locked table /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS/POLARIZATION: 4 columns, 1 rows
Successful readonly open of autonoread-locked table /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS/DATA_DESCRIPTION: 3 columns, 1 rows
Successful readonly open of autonoread-locked table /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS/SPECTRAL_WINDOW: 14 columns, 1 rows
Successful readonly open of autonoread-locked table /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS/FIELD: 9 columns, 1 rows
Using LSM module from Tigger (using svn version) at /usr/lib/python2.7/dist-packages/Tigger (in path)
Loading test-lsm1.lsm.html: Tigger sky model
========== Simulating MODEL_DATA
========== Imaging MODEL_DATA
MSUtils: imager args are python /usr/lib/python2.7/dist-packages/Cattery/Meow/make_dirty_image.py data=MODEL_DATA ms=WSRT.MS mode=channel weight=radial stokes=IQUV npix=1024 prefervelocity=False cellsize=14.0625arcsec spwid=0 field=0 padding=2.000000 image_viewer=none filter=12.000000arcsec,12.000000arcsec,0.000000deg chanmode=channel nchan=64 chanstart=0 chanstep=1 img_nchan=1 img_chanstart=0 img_chanstep=64 fits=WSRT.MS.MODEL_DATA.channel.1ch.fits image=WSRT.MS.MODEL_DATA.channel.1ch.img remove_image
Column MODEL_DATA not added; it already exists
added column CORRECTED_DATA
added column IMAGING_WEIGHT
defined keyword CHANNEL_SELECTION in column MODEL_DATA
lwimager: Version 20130816-OMS
2014-02-20 13:16:03 INFO Imager::open() Opening MeasurementSet /home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS
2014-02-20 13:16:03 INFO imager::data selection Selecting data
2014-02-20 13:16:03 INFO imager::data selection Selected 6552 out of 7560 visibilities.
2014-02-20 13:16:03 INFO Imager::selectDataChannel() Selecting 64 channels, starting at 0, stepped by 1, for spw 0
2014-02-20 13:16:03 INFO imager::defineimage() Defining image properties:nx=1024 ny=1024 cellx='14.0625arcsec' celly='14.0625arcsec' stokes=IQUV' mode=channel nchan=1 start=0 step=64 spwids=[0] fieldid=0 facets=1 frame=1 distance='0m'
2014-02-20 13:16:03 INFO imager::defineimage() phaseCenter='field-0 ' mStart='Radialvelocity: 0' qStep='1 km/s'' mFreqStart='Frequency: 0
2014-02-20 13:16:03 INFO imager::weight() Weighting MS: Imaging weights will be changed
2014-02-20 13:16:03 INFO imager::weight() Radial weighting
2014-02-20 13:16:03 INFO imager::filter() Imaging weights will be tapered
2014-02-20 13:16:03 INFO VisImagingWeight::setFilter() Filtering for Gaussian of shape: 12 by 12 (arcsec) at p.a. 0 (degrees)
2014-02-20 13:16:03 INFO imager::setoptions() Setting processing options
2014-02-20 13:16:03 INFO imager::makeimage() Calculating image (without full skyequation)
2014-02-20 13:16:03 INFO imager::makeimage() Making dirty image from model data
2014-02-20 13:16:03 INFO imager::makeimage() Image is : WSRT.MS.MODEL_DATA.channel.1ch.img
2014-02-20 13:16:03 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Did not get the position of LOFAR from data repository
2014-02-20 13:16:03 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Please contact CASA to add it to the repository.
2014-02-20 13:16:03 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Frequency conversion will not work
2014-02-20 13:16:03 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657) Leap second table TAI_UTC seems out-of-date.
2014-02-20 13:16:03 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657)+ Until table is updated (see aips++ manager) times and coordinates
2014-02-20 13:16:03 SEVERE MeasTable::dUTC(Double) (file /build/buildd/casacore-1.5.0+svn21411/measures/Measures/MeasTable.cc, line 6657)+ derived from UTC could be wrong by 1s or more.
2014-02-20 13:16:03 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Did not get the position of LOFAR from data repository
2014-02-20 13:16:03 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Please contact CASA to add it to the repository.
2014-02-20 13:16:03 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Frequency conversion will not work
2014-02-20 13:16:04 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Did not get the position of LOFAR from data repository
2014-02-20 13:16:04 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Please contact CASA to add it to the repository.
2014-02-20 13:16:04 WARN Imager::imagecoordinates() (file /build/buildd/lwimager-1.3.0/synthesis/MeasurementEquations/Imager2.cc, line 231) Frequency conversion will not work
2014-02-20 13:16:04 INFO imager::createFTMachine() Performing interferometric gridding...
2014-02-20 13:16:05 INFO ImageFitsConverter::ImageToFITS Copying '/home/gijs/build/meqtrees-cattery/test/Batchtest/WSRT.MS.MODEL_DATA.channel.1ch.img' to 'WSRT.MS.MODEL_DATA.channel.1ch.fits' Copying plane by plane (1048576 pixels).
lwimager normally ended
Image WSRT.MS.MODEL_DATA.channel.1ch.fits ready. No image viewer selected (see TDL Exec menu).
========== Verifying test image
Stopping meqserver
Traceback (most recent call last):
File "batch_test.py", line 126, in <module>
verify_image('WSRT.MS.MODEL_DATA.channel.1ch.fits',path('test-refimage.fits'),maxdelta=1e-3);
File "batch_test.py", line 38, in verify_image
raise RuntimeError,"%s and %s differ by %g"%(file1,file2,delta);
RuntimeError: WSRT.MS.MODEL_DATA.channel.1ch.fits and /home/gijs/build/meqtrees-cattery/test/Batchtest/test-refimage.fits differ by 0.00134248
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment