========================================= PC Lafter Developer Documentation Version 1.0b Created: July 29, 1997 by Nilesh Agarwalla Last Updated: August 12th, 1997 by Nilesh Agarwalla (c) Massachusetts Institute of Technology ========================================= Index ----- I - System Requirements II - General Architecture and Design III - PC Lafter Classes IV - Known Bugs V - History VI - Appendix A. SONY EVI-D30 to PC cable wiring diagram B. Contact Information ----------------------------------- I - System Requirements ----------------------------------- CPU: Dual (or higher) Pentium Pro processors RAM: 64 MB (128 MB Recommended) SERIAL: COM Port (RS-232) free VIDEO: At least 4 MB high end PCI video card DEPTH: 32bit True Color mode RESOL: 1024x768 (1280x1024 Recommended) CAMERA: Sony EVI-D30 Remote Controllable Camera CABLES: DB9F to DIN8M cable (See Appendix A for wiring diagram) S-Video cable CAPTURE: Coreco TCI video capture board OS: Windows NT 4.0 IDE: Visual C++ 5.0 Note: Software was developed on a Dual Pentium Pro 200 system with 128 MB RAM and a Matrox Millenium 8MB video card. Works with a Dual Pentium setup, but not recommended. ----------------------------------------------- II - General Architecture and Design ----------------------------------------------- PC Lafter was developed in Microsoft Visual C++ 5.0 under the Windows NT 4.0 operating system. PC Lafter utilizes the multiprocessor capabilities of Windows NT. A. Threads There are four main threads that are interdependent and must run simultaneously: the main program, the grabber, the camera control, and the vision. Main Program: This thread controls the user interface and the display of all the dialogs and live windows. Grabber: This thread communicates with the video capture board and constantly updates memory with the new images. Camera Control: This thread communicates with the video camera and constantly sends update commands to the camera. Vision: This thread is the most processor intensive and utilizes one of the multiple processors on its own. This thread depends on the Grabber and Camera Control threads being active and working properly because images obtained from the Grabber are analyzed and then appropriate commands are sent to the Camera Control thread. -------------------------------------------------- III - PC Lafter Classes (important ones) -------------------------------------------------- A. Class Hierarchy ------------------ CVision VTciGrabber CDialog \CLafterDlg CDialog \CHistogramDialog CDialog \CTrackerDialog CDialog \CDisplayDialog \CStaticFrameDialog CDialog \CDisplayDialog \CStaticFrameBBDialog \CGrabbedImageDialog CStatic CDisplay \CStaticFrame \CStaticFrameBB B. Descriptions of classes -------------------------- VTciGrabber ----------- Constructor: default Initializer: Init(CStringList *msgs = 0, int res_factor = 2) msgs // for debug purposes (MUST BE NULL) res_factor // reduction factor to shrink images Data Members public: // Shrinks image ErrorNumber DirtyReduce(TBitMap &dest, int factor = 2); dest // reference to Bitmap image to shrink factor // reduction factor // Starts Grabber Thread ErrorNumber StartGrab(HWND hWnd, CListBox *lstBox, CDisplay *disp, BOOL display); hWnd // handle to parent window lstBox // for debug purposes (MUST BE NULL) disp // pointer to CDisplay that stores image display // True if image is to be displayed void StopGrab(); // Stops Grabber Thread void Snap(CDisplay *disp); // Grabs a still image into disp disp // pointer to CDisplay object int Width(); // returns Width of image int Height(); // returns Height of Image int PixelSize(); // returns PixelSize of Image BOOL Grabbing(); // returns TRUE if Grabber Thread running float FrameRate(); // returns frame/sec void Dispose(); // Destroys Grabber object friend UINT GrabThread(LPVOID param); protected: HGLOBAL hBuf; // Image Buffer HANDLE m_Thread; // Handle of grabber thread ThreadParams params; // Grabber Thread parameters BOOL m_bGrabbing; // TRUE if Grabber Thread running LPVOID m_lpBits; // pointer to image bits CEvent stopEvent; // Signals grabber Thread to stop int m_nPixSize; // PixelSize of Image int m_nWidth; // Width of image int m_nHeight; // Height of image void * ImgBuf2; // Image buffer #2 void * ImgBuf1; // Image buffer #1 CVTimer m_timer; // Timer object int m_nFramesGrabbed; // Hold number of frames grabbed Description: See description of Threads above. CVision ------- Constructor: default Initializer: Init(VTciGrabber *grabber, int factor) grabber // Grabber object (must be initialized) factor // Division factor for reduced width images Data Members public: // Starts vision thread void StartVision(CStaticFrameBBDialog *pEstim, CHistogramDialog *pHisto, CTrackerDialog *pTracker, CStaticFrameDialog *pConnex); pEstim // pointer to Estimation Dialog pHisto // pointer to Histogram Dialog pTracker // pointer to Tracker Dialog pConnex // Pointer to Connex Dialog void StopVision(); // Stops vision Thread void Dispose(); // destroys vision object int Factor(); // returns reduction factor BOOL VisionOn(); // returns TRUE if vision thread running CRITICAL_SECTION m_Sem; // for thread synchronization friend UINT VisionThread(LPVOID param); // Vision Thread protected: CWinThread* pTh; // pointer to Vision thread HANDLE m_Thread; // handle to Vision thread VisionThreadParams params; // Vision Thread parameters CEvent stopEvent; // signals Vision thread to stop TBitMap m_bm; // Live image bitmap TBitMap m_histoBM; // Histogram image bitmap TBitMap m_connexBM; // Connexity image bitmap VTciGrabber *m_pGrabber; // pointer to grabber int m_dFactor; // reduction factor BOOL m_bVisionOn; // True if vision thread is running Description: See description of Threads above. CDialog \CLafterDlg ---------- Constructor: CLafterDlg(CWnd* pParent = NULL); pParent // Window pointer of parent Initializer: Called by Windows Subsystem automatically Data Members: private: // The following are the dialog object pointers CStaticFrameDialog* m_pLiveDlg; CStaticFrameBBDialog* m_pEstimationDlg; CStaticFrameDialog* m_pFaceConnectDlg; CHistogramDialog* m_pHistogramDlg; // Vision Object CVision m_vision; public: // The following are TRUE if the appropriate dialog // is visible. Otherwise FALSE. BOOL m_bEstimation; BOOL m_bFaceHisto; BOOL m_bLive; BOOL m_bFaceConnect; protected: // Grabber Object VTciGrabber grabber; Description: This is the main dialog from which all the children are created. All of the child dialog objects are created in the constructor of this dialog and are modeless dialogs. The dialog objects are always present for the duration of the program, so public data members in the children can be accessed by the parent at any time during runtime. The Grabber, Vision, and Dialog objects are initialized in OnInitDialog(). A Timer is set for the updating of the Frame Rate on the main interface. When a button is pressed on the interface, the appropriate window is shown or hidden. On closing, the grabbing and vision threads are terminated and then the dialog objects are destroyed. CDialog \CHistogramDialog ---------------- Constructor: CHistogramDialog(CDialog* pDlg) pDlg // Dialog Pointer of Parent used to post messages to parent of window status (i.e. visible/hidden) Initializer: BOOL Create(int nPixSize, int nWidth, int nHeight, VTciGrabber *pGrabber, int nInitThreshold, float fInitMaxDist) nPixSize // PixelSize of the embedded histogram image nWidth // Width of the embedded histogram image nHeight // Height of the embedded histogram image pGrabber // pointer to grabber object nInitThreshold // initial Threshold value for dialog fInitMaxDist // initial MaxDist value for dialog Data Members: private: CGrabbedImageDialog *m_pGrabbedImgDlg; // Dialog pointer BOOL m_bGrabbedImgDlg; // True if visible BOOL m_bFirstGrabbedImgDlg; // True if first VTciGrabber *m_pGrabber; // Grabber Pointer THistogram m_histo; // Histogram object int m_nSliderThreshold; // Dialog control variable int m_nArrayThreshold[256]; // Dialog control display var CPoint m_pointGrabbedImgDlg; // Position of GrabbedImgDlg CDialog* m_pDlg; // pointer to parent dialog int m_nFrameHeight; // height of embedded frame int m_nFrameWidth; // width of embedded frame BITMAPINFO_8 m_stdBmInfo; // bitmap info public: int m_nThreshold; // Dialog control variable CStaticFrame m_picProcessedImg; // embedded image int Hide(); // Hides Window int Show(); // Shows Window THistogram* Histo(); // Returns histgram object CRITICAL_SECTION* Sem(); Returns Critical Section int Threshold(); // Returns threshold value protected: // The following are data members which relate to // controls on the dialog float m_fMaxDist; float m_fResolution; int m_nColorSource; BOOL m_bRobust; CRITICAL_SECTION m_Sem; // Used in thread sychronization Description: This is the dialog where the main vision parameters are set. You can recalculate the probability maps to tweak the vision process. A child dialog is spawned from this dialog if a bitmap is used recalculate the probability map. This new dialog is of class CGrabbedImageDialog and is discussed later. CDialog \CTrackerDialog ---------------- Constructor: CTrackerDialog(CDialog* pDlg) pDlg // Dialog Pointer of Parent used to post messages to parent of window status (i.e. visible/hidden) Initializer: Init() Data Members: public: float m_fHeightRatio; // Dialog control var. float m_fTheoHeightRatio; // Dialog Control var. float m_fPError; // Dialog Control var. float m_fP; // Dialog Control var. int m_nD; // Dialog Control var. int m_fDError; // Dialog Control var. int m_nPanThreshold; // Dialog Control var. int m_nTiltThreshold; // Dialog Control var. float m_fHeightThreshold; // Dialog Control var. float m_fTheoHeightThreshold; // Dialog Control var. int m_nNumFrames; // Dialog Control var. float m_fArrayHeightRatio[101]; // Dialog Display var. float m_fArrayTheoHeightRatio[101]; // Dialog Display var. float m_fArrayP[101]; // Dialog Display var. float m_fArrayHeightThreshold[101]; // Dialog Display var. float m_fArrayTheoHeightThreshold[101]; // Dialog Display float m_fXErr; // Dialog Display var. float m_fYErr; // Dialog Display var. float m_fPanSpeed; // Dialog Display var. float m_fTiltSpeed; // Dialog Display var. float m_fZoomRatio; // Dialog Display var. BOOL m_bPanic; // Dialog Display var. private: int m_nSliderHeightRatio; // internal control var. int m_nSliderTheoHeightRatio; // internal control var. int m_nSliderP; // internal control var. int m_nSliderHeightThreshold; // internal control var. int m_nSliderTheoHeightThreshold; // internal control var. TBoxTracker m_tracker; // Camera Tracker object CRITICAL_SECTION m_Sem; // for thread synchronization CDialog* m_pDlg; // pointer to parent dialog public: BOOL Create(); // Creates Window object TBoxTracker* BoxTracker(); // Returns tracker object pointer CRITICAL_SECTION* Sem(); // for thread synchronization // Sets paramaters in tracker object void SetParameters(CRect* bounds = (CRect*)NULL, float heightRatio = -1, float p = -1, float d = -1, float xThreshold = -1, float yThreshold = -1, float ratioThreshold = -1, float theoRatio = -1, float theoRatioThreshold = -1); int Hide(); // Hides Window int Show(); // Shows Window Description: This dialog holds and allows the user to set various parameters pertaining to the box tracker limits. This is used to fine tune the vision process. The Vision thread has a pointer to this tracker dialog which is why a Critical Section is needed. CDisplay -------- Constructor: CDisplay() Initializer: Create(BITMAPINFO_8 &bmInfo, BOOL bMatchScreen) bmInfo // Structure holding parameters for the bitmap bMatchScreen // ??? Data Members: public: static short defaultBorderWidth; VDib m_dib; // bitmap object CSize m_size; // size of bitmap TVDibValidity m_bValidDib; // TRUE of Dib is valid // links bitmap object to image bits in memory virtual void AttachMemToDib(LPVOID bits, BOOL bAlwaysValid = FALSE); Description: Base class to show a VDib object on the screen. This is a virtual class and should be derived. CStatic CDisplay \CStaticFrame ------------ Constructor: CStaticFrame(CDialog* pDlg) pDlg // Dialog Pointer of parent Initializer: Create(BITMAPINFO_8 &bmInfo, BOOL bMatchScreen) bmInfo // Structure holding parameters for the bitmap bMatchScreen // ??? Data Members: private: CDialog *m_pDlg; // pointer to parent dialog public: void Refresh (); // Redraws Dib // Centers Dib in Parent Window virtual void CenterWindow( CWnd* pAlternateOwner = NULL ); Description: An object of this class can be placed in a window frame which will give you a seamlessly integrated still or live frame within a window. CStatic CDisplay \CStaticFrame \CStaticFrameBB -------------- Constructor: CStaticFrameBB(CDialog* pDlg) pDlg // Dialog Pointer of parent Initializer: Create(BITMAPINFO_8 &bmInfo, BOOL bMatchScreen, BOOL bDrawBB, BOOL bDrawEig) bmInfo // Structure holding parameters for the bitmap bMatchScreen // ??? bDrawBB // True if BoundingBox drawn bDrawEig // True if Eigen Crosses drawn Data Members: private: CBrush m_brush; // brush object used to draw BB CPen m_pen; // pen object used to CRect m_rect; BOOL m_bDrawBB; BOOL m_bDrawEig; TPointArray m_arrEigPoint; public: void SetRect(CRect &rect); // void SetEigPoints(TPointArray &arrEigPoint); void SetDrawBB(); void SetDrawEig(); void UnsetDrawBB(); void UnsetDrawEig(); Description: Adds Bounding Box and Eigen Cross overlaying on image CDialog \CDisplayDialog -------------- Constructor: CDisplayDialog(CDialog* pDlg, int id = 0 ) pDlg // Dialog Pointer of Parent used to post messages to parent of window status (i.e. visible/hidden) id // Unique ID used in communicating with parent dialog Initializer: Create(int nPixSize, int Width, int Height) nPixSize // PixelSize of the embedded image Width // Width of the embedded image Height // Height of the embedded image Data Members: public: virtual BITMAPINFO_8 BmInfo(); // returns BMINFO structure virtual void CenterBitmap( ); // Centers bitmap in Window int ID(); // Returns ID void Hide(); // Hides Window void Show(); // Shows Window Description: Base class for the main StaticFrame and StaticFrameBB dialogs. Handles the creation, showing, and hiding of the dialog. CDialog \CDisplayDialog \CStaticFrameDialog ------------------ Constructor: CStaticFrameDialog(CDialog* pDlg, int id = 0); pDlg // Dialog Pointer of Parent used to post messages to parent of window status (i.e. visible/hidden) id // Unique ID used in communicating with parent dialog Initializer: BOOL Create(int nPixSize, int Width, int Height); nPixSize // PixelSize of the embedded image Width // Width of the embedded image Height // Height of the embedded image Data Members: public: CStaticFrame m_picLiveImg; // embedded image Description: This is a wrapper for the CStaticFrame image. It's main purpose is to add message passing capabilities. CDialog \CDisplayDialog \CStaticFrameBBDialog -------------------- Constructor: CStaticFrameBBDialog(CDialog* pDlg, int id = 0); pDlg // Dialog Pointer of Parent used to post messages to parent of window status (i.e. visible/hidden) id // Unique ID used in communicating with parent dialog Initializer: BOOL Create(int nPixSize, int Width, int Height, BOOL bMatchScreen, BOOL bDrawBB, BOOL bDrawEig); nPixSize // PixelSize of the embedded image Width // Width of the embedded image Height // Height of the embedded image bMatchScreen // ??? bDrawBB // True if BoundingBox drawn bDrawEig // True if Eigen Crosses drawn Data Members: public: CStaticFrameBB m_picLiveImg; // embedded image Description: This is a wrapper for the CStaticFrameBB image. It's main purpose is to add message passing capabilities. CDialog \CDisplayDialog \CStaticFrameBBDialog \CGrabbedImageDialog ------------------- Constructor: CGrabbedImageDialog(CWnd* pDlg); pDlg // Window pointer of parent Initializer: see CStaticFrameBBDialog Data Members: public: BOOL m_bFirstBox; // True if first box CRect GetBoundingBox(); // Returns CRect of frame Description: Displays a still image on the screen. Allows a user to frame part of the image using the mouse. GetBoundingBox() returns the CRect of the user defined frame. This is used in the Histogram Dialog. The Grab button is pressed on the Histogram Dialog to grab a new video image into the the window. -------------------------- IV - Known Bugs -------------------------- * Windows do not refresh properly when they are overlapping Temporary fix: click on the title bar to refresh affected window. * When you try to save to a file that already exists in the Histogram SAVE Dialog, the program seems to lock. What is happening is the MessageBox asking for verification for overwriting the file is created behind the SAVE dialog. Temporary fix: press the ALT key to pop the confirmation on top * The Camera doesn't initialize properly, thus causing the program to lock up. Temporary fix: turn camera on and off ----------------------- V - History ----------------------- 08/15/97 - PC Lafter v1.0b released ------------------------ VI - Appendix ------------------------ ------------------------------------------ A. SONY EVI-D30 to PC cable wiring diagram ------------------------------------------ A readily available cable doesn't seem to exist for our application, so you will probably be required to create your own cable which interfaces the Sony EVI-D30 camera to the Serial DB9 port of a PC. You will need the following: (1) DB9F connector (PC Serial) (1) DIN8M connector (MAC Serial) 7 wire cabling Note: If you are unable to find the connectors, you can buy a MAC PASS-THRU Serial Cable w/ Male connectors and a PC PASS-THRU DB9 Serial Extension. You can cut the wires in half and splice the proper ends together. You must connect the pins as follows: PIN #'s /-------------\ | DB9F | MD8M | |------+------| | 9 | 1 | | 5 | 2 | | 2 | 3 | | 7 | 4 | | 3 | 5 | | 4 | 6 | | 8 | 7 | \-------------/ Pins 1 and 6 on the DB9 are unused. Pin 8 on the MD8 is unused. ---------------------- B. Contact Information ---------------------- Nuria Oliver MIT Media Lab nuria@media.mit.edu Vasanth Philomin University of Maryland vasi@cs.umd.edu Nilesh Agarwalla MIT Media Lab nilesh@mit.edu Coreco Inc. 6969 Trans-Canada Highway, Suite #142 St. Laurent, Quebec H4T-1V8 Canada www.coreco.com Sony ??? Matrox Graphics Inc. 1055 St. Regis blvd. Dorval, Quebec Canada H9P 2T4 www.matrox.com /*************************************************************************** * * Copyright 1997 by the Massachusetts Institute of Technology. All * rights reserved. * * Developed by the Perceptual Computing Section * at the Media Laboratory, MIT, Cambridge, Massachusetts. * * Permission to use, copy, or modify this software and its documentation * for educational and research purposes only and without fee is hereby * granted, provided that this copyright notice and the original authors's * names appear on all copies and supporting documentation. If individual * files are separated from this distribution directory structure, this * copyright notice must be included. For any other uses of this software, * in original or modified form, including but not limited to distribution * in whole or in part, specific prior permission must be obtained from * MIT. These programs shall not be used, rewritten, or adapted as the * basis of a commercial software or hardware product without first * obtaining appropriate licenses from MIT. MIT. makes no representations * about the suitability of this software for any purpose. It is provided * "as is" without express or implied warranty. * **************************************************************************/