v1_1_3 (May 19, 2003)

- cosmic-ray handling has been altered: CRs found in 2d are excluded
  when the data is coadded (rather than just using the temporal
  variation to find them in the coaddition process), and INFOMASK bit
  2^3 in spslit structures indicates where cosmic rays have been found
  (minor speedups result).  This should yield improvements in
  CR-rejection particularly for cases  with only 2 frames of data.

- We had been under the impression that the TOPDIST/BOTDIST bug (they
  are inverted from their original definition) in Drew Phillips's mask
  design software was fixed in mid-October, so the pipeline assumed no
  fix was needed for masks created after that date.  Instead, Drew
  opted to leave his code untouched; this means that the pipeline
  assumed objects were in the wrong position for recently-designed
  masks (flipped about the center of the slit), with bad consequences
  for object extraction, sky subtraction, etc.  Masks designed after
  mid-October and run through the pipeline will need to be rerun.

- "quicklook" data verification routines have been added.  For the
  moment, at least, they are DEEP- specific.  Missing species in arcs,
  FCS misalignment/buckling, alignment offsets in the spatial
  direction, bad or degrading seeing, and signal-to-noise are all
  evaluated, and audible & pop-up warnings are issued to the user if
  there are problems.  /QUICK keywords have been added to many of the
  pipeline routines; setting the keyword will cause the routines to
  act in less robust, but faster, ways.

- obslog routine added to generate log info on science files in DEEP2
  logsheet form.  typing 'obslog' at a command prompt in the desired
  directory  yields a log for all files; 'obslog n', where n is an
  integer, gives log info on the last n files.

v1_1_2 (April 14, 2003)

- Linelist has been tested for a variety of 600-line scenarios, and
  tweaked accordingly.

- Previously, CR's on arcs could cause the arc tilt as a function of
  wavelength to have bad fits.  Two changes to deimos_arcfit have been
  added to prevent this; hopefully, we will no longer suffer from bad tilts!

- a color table issue with qa_check has been fixed.

--------------------------------------------------------------------
v1_1_1 (April 11, 2003)

- A number of minor bug fixes have occurred; notably, SLITPA and
  MASKPA in v1_1 file headers were incorrect.

- Zn has been added to the non-DEEP linelist (lamp_NIST_blue.dat).  A
  few lines might be helpful in the extreme blue, though ~10-20s
  exposures might be desirable for picking them up.

--------------------------------------------------------------------

v1_1 (April 1, 2003)


Major changes:


- Drew Phillip's optical model is now used for initial wavelength
  guesses.  Large tweaks are sometimes still required (e.g. due to
  differences in FCS setup from the model), but nevertheless this has
  proved to be quite successful.

- Arc handling has been totally revamped, allowing good solutions for
  600-line data; among other changes, a new lamp list,
  lamp_NIST_blue.dat, should work better than the default
  lamp_NIST.dat for low-resolution/non-DEEP2 data.  We can now deal
  with multiple arcs (including combining partially saturated with
  unsaturated arcs; see COMBINE_ARCS), and can use separate arcs for
  red side vs. blue.

- Slits overlapping edges of chips are now reduced, not ignored.  This
  could add as many as 8 objects/mask (~2-6 should be typical).
  Regions with no data have inverse variance 0 and the 2^2 bit of MASK
  set to 1.

- The derivative tails at the spatial ends of slits (due to something
  like sub-pixel shifts between the slitfunction derived from flats
  and the science data, though that cannot be the actual cause) are
  now corrected when possible, giving us ~2 more rows per slit for sky.

- Non-local sky subtraction has been greatly improved, and is now used
  throughout the pipeline (including extractions & spec1d, with
  non-local results in separate HDUs).

- B-spline breakpoint handling and sky region determination has
  changed, yielding much better (though still noisy) results for short slits.

- Extractions totally revamped.  Standard extractions are no longer a
  boxcar with interpolation and a non-inverse variance weighted
  optimal extraction, but instead a boxcar with bad pixels compensated
  for based on the fraction of light from the object missing (using
  the object's profile) and an optimal extraction with inverse
  variance weighting.  A major bug (which caused the
  optimal-extraction kernel to be the same for all objects) has been fixed.

- Sky spectra are now stored in the spec1d structures, and the inverse
  variance in the extracted spectra due to covariant sky subtraction
  is properly included.

- Positions of serendipitous sources on the sky (RA/dec) are now
  determined relative to object position using slit PA.  Serendips now
  have naming convention 'sXXXXXXX[b/c/d/...]' where XXXXXXXX is the label of
  a source on the same slit, and [b/c/d/...] is a letter, b for the
  first serendip on a slit, c for the second, etc.

- Flat-fielding, and particularly slit function determination, has
  been made much more robust; spectral ends of chips are also much 
  better than before. Additionally, we have greatly improved the behavior of
  the b-spline at the spectral ends of the array.

- To deal with the time variations in the FCS scattered light, the pipeline
  now chooses amongst multiple superdarks rather than always using an
  August one.

- Bad pixel mask has been greatly improved (using superdark data).

- Old QA routines have been updated and replaced with QA_CHECK .


More minor changes and fixes:


- More keywords saved in FITS headers (including Gaussian sigma of
  skylines and PAs for rotation curve analysis).

- The SKYIND tag in spslit files has been removed to prevent
  occasional problems combining spslit structures.

- spec1d structures have been changed (e.g. to include the sky
  spectra); see documentation.  Extraction width definitions have been
  unified between boxcar and optimal; the IVARFUDGE parameter is an
  estimate of how much one might want to multiply the stated IVAR by
  to match the actual fluctuations in the extracted spectrum (this was
  added mostly as a diagnostic; it varies significantly from one only
  for sky slitlets [due to using the same rows for the sky bspline and
  extraction], bright objects [which violate the assumptions made],
  and short slits with concomitantly noisy sky subtraction (for which
  IVARFUDGE reflects the reality).

- Arc lines are now identified as blends based on what lines are
  actually in a mask's arcs, not based on the complete linelist.

- New .plan file keywords: REDARC, BLUEARC, and LINELIST .

- SPEC1D_ASCII routine to convert spec1d output to ASCII files.

- FLAG_CR now finds CR's iteratively (only used for bsplines)


To be done:

- When we have few identified arc lines, if we fit, say, 2 terms in
  the wavelength solution, fix the 3rd-6th from the optical model and
  fit the first two, rather than only including 2 terms in the
  solution total.

- Develop the ability to combine results from different arc frames,
  rather than combining all frames and then fitting.

- Handle the varying resolution along a slit that comes from big
  notches in the slit function.  The interim fix might be to only use
  sky rows with the same slitfn value as the object; longer-term, we
  might convolve all rows to matching resolution.

- Incorporate atmospheric dispersion and wavelength-dependent seeing
  into the extractions.  This will probably require a rewrite &
  generalization of extract1d.

- Flux calibration.

- Re-implement quicklook.

- Frame-by-frame QA diagnostics.

- Time-varying pixflat files, not just superdarks

------------------------------------------------------------------------------

Changes from v1_0 -> v1_0_2 (November 2002):

- Much better handling of vignetting added

- flat fields, esp. 2d slit functions and fringe amplitude, improved.

- minor arcfitting improvements.

------------------------------------------------------------------------------
v1_0 (Nov. 7, 2002)

Changes:

- nonlocal sky is now implemented in the pipeline! (more tuning is
  still required, though.)  For slits with <12 rows suitable for
  measuring the sky spectrum, a 2d, nonlocal-sky-subtracted spectrum 
  can be found in HDU 3 of the slit.XXXX file (no differences in
  calibslit or spSlit files).  
- Extracted 1d spectra based on the nonlocal version of the slit will 
  then be found in HDUs 5-8 of the spec1d.XXXX file.  
- Allslit files (nonzero only for those slits where nonlocal sky was
  performed, to facilitate compression) are also created, named
  Allslits[0 or 1].XXXX.nonlocal.fits.gz

- an array named INFOMASK has been added (at the end) to the spSlit
  and slit BINTABLEs.  This array is like MASK, but indicates
  non-catastrophic problems.  The 2^0th bit is set for all >0% vignetted
  pixels, and the 2^1 bit for pixels identified as cosmic rays in 2d
  (with no reference to other frames).

- The bar between slitlets in Allslit files has been enlarged, and now
  is set to -1000 - slitnum, where slitnum is the slitnumber of the
  slit _below_ the bar.

- serendip detection is greatly improved - more robust to outlying
  pixels and background gradients, reducing the number of bogus
  serendips greatly.

- extractions have been tested with fake 2d spectra inserted on sky
  slits and extraction widths optimized (1.1*FWHM for boxcar, 
  +/- 1.75*sigma [1.5*FWHM] for optimal; optimal had definitively better S/N

- QA plots (quality_plots.ps in the /ps subdirectory of each mask)
  have been improved.  A robust estimate of the S/N per pixel in an
  extracted spectrum for an R=23.5 object is obtained for each mask,
  allowing mask-to-mask comparisons.

- all (>0%) vignetted pixels on the red side are now masked
- scattered light is now removed before the vignetting flat is
  created.  This yielded only minor improvements.
- improved vignetting mask in deimos_badchip.
- improved determination of DLAM when arc lines cross the vignetting 
transition


Problems:
- insufficient number of sky rows going into the bsplines on short
  slits, causing artifacts - need nonlocal sky to be optimized.
- slit edge fix not incorporated into the pipeline yet, though we
  think we know how
- wavelength solutions still have problems at sub-pixel level on
  alignment stars
- need to look at 4249 Allslit file again when regenerated --
  looks like a bad column was not fully masked?
- should make inverse-variance weighted extractions an option
- need to check if slit number bars in Allslit files are always right
- serendip identification needs more tuning, need to implement
  matching to PCAT catalogs
- more tests of extractions and QA assessments

------------------------------------------------------------------------------