Notice: MediaWiki has been updated. Report any rough edges to marcan@marcan.st
Protocol Documentation: Difference between revisions
(→Control Packet Structure: Fix link) |
|||
Line 97: | Line 97: | ||
uint8_t _frame[422400]; | uint8_t _frame[422400]; | ||
This is a 12-byte struct followed with a data array and forms the basis of the Kinect's depth/color camera perception protocol. As you can tell we definitely have some magic inside this structure. Some of them are most likely control/status commands and will be determined later on. The _frame_pos and _frame are private member variables which work to effectively process the data. | This is a 12-byte struct followed with a data array and forms the basis of the Kinect's depth/color camera perception protocol. As you can tell we definitely have some magic inside this structure. Some of them are most likely control/status commands and will be determined later on. The _frame_pos and _frame are private member variables which work to effectively process the data. | ||
Line 144: | Line 144: | ||
</code> | </code> | ||
Looks fairly routine, but I'll be the first to admit bit-wise operations make my head hurt, doubly so for efficient bit-wise operations. | Looks fairly routine, but I'll be the first to admit bit-wise operations make my head hurt, doubly so for efficient bit-wise operations. | ||
== RGB Camera == | == RGB Camera == |
Revision as of 17:29, 17 November 2010
OpenKinect Collaboration
Please check back often, as additional insight and knowledge is gained it will be documented here.
Links
- https://github.com/adafruit/Kinect/blob/master/usbmotor.py - Python motor control code - does the "move up and down" action.
- https://gist.github.com/670533 - Sample python code
- https://github.com/adafruit/Kinect/tree/master/USBlogs/ - Some USB logs, curtosy of adafruit, hosting on the wonderful large binary file distribution site, github, for easy downloading.
USB Communication
USB Communicatoin is currently under development, with most communication taking place using pyusb, and libusb.
Devices
- 04 - NUI Motor
- 07 E1 - NUI Camera - RGB
- 07 E2 - NUI Camera - Depth
- 06 - NUI Audio
Control Packet Structure
All motor/LED commands are sent via control transfers. More information on the basic structure of these packets at http://www.beyondlogic.org/usbnutshell/usb6.shtml#SetupPacket
Control Transfer (8-bytes) Request: Type (1 byte) Request (1 byte) Value (2 bytes) Index (2 bytes) Length (2 bytes)
For read packets (Type & 0x80 ?) length is the length of the response.
Motor Initialization
Verify readiness? Send: 0xC0, 0x10, 0x0000, 0x0000, 0x0001 Response: 0x22
Enable motor control? Write 0x40, 0x6, 0x0001, 0x0000, 0x0000
Setting Position
Send: 0x40, 0x31, 0xFFXX, 0x0000, 0x0000 Response: ??
where XX is: A valid motor range between 0xFF to ~0xC0, with 0xFF being level to the horizon.
Upon receiving the command the motor will drive itself to the specified level and stop. No need to send successive levels. Approximate drive time is 1-1.5 seconds.
Warning: Sending the motor past 0xC0 has been shown to cause stalling of the motor. There are no built-in safe-gaurds to prevent this kind of command from being sent. Due to the low torque and power consumption no damage is likely but this should be avoided to ensure maximum component life.
Kinect LED Color
The LED on the front of the Kinect can be directly controled with a control transfer packet.
Send: 0x40 0x06 0x00XX 0x0 0x0 where XX is one of the following values:
Val - Function
- 0x00 - OFF
- 0x01 - GREEN
- 0x02 - RED
- 0x03 - YELLOW
- 0x04 - Blink Green
- 0x05 - Blink Green
- 0x06 - Fast Alternate Red Yellow
- 0x07 - Fast Alternate Red
Values greater than 0x08 only use the last 3 bits.
Cameras
The depth camera returns values with 11-bits of precision.
RGB data should follow similar conventions but will need to be analyzed. RGB frames are significantly bigger and encoded using a Bayer pattern.
Relevent Bits of Code
The most important piece of data gathered from the libkinect project is the following struct:
struct __attribute__ ((packed)) frame_info { uint8_t magic[2]; //"RB" uint8_t control; // 00 - means incoming data, other values (if any) are control codes uint8_t cmd; // if control=0, 71 or 81- new frame, 72 or 82- current frame, 75 or 85 - EOF (7x-depth, 8x-color) uint8_t SeqNum; uint8_t pkt_seq; uint8_t unk_06; // 04 on EOF uint8_t unk_e0; // 78 on EOF uint32_t time_stamp; // all 0 on new frame. packets for one frame has the same timestamp uint8_t data[]; }; int _frame_pos; uint8_t _frame[422400];
This is a 12-byte struct followed with a data array and forms the basis of the Kinect's depth/color camera perception protocol. As you can tell we definitely have some magic inside this structure. Some of them are most likely control/status commands and will be determined later on. The _frame_pos and _frame are private member variables which work to effectively process the data.
To process an image we do the following, after pushing the data into the above struct:
- If cmd == 0x71 we have reached the end of the current frame and need to go onto the next frame. Set our _frame_pos pointer to 0.
- We get the length of the data array. This is the total length of the packet subtracting the length of the header information in the above struct.
- We memcpy the data array into our _frame array, using _frame_pos as an base index position.
- If cmd == 0x75 we have reached the end of the current frame, time to shoot it to a output of some sort.
There are 242 packets for one frame for depth camera (including 0x71 and 0x75 packets). All packets are 1760 bytes except 0x75 packet - 1144 bytes. Minus headers it gives 422400 bytes of data.
There are 162 packets for one frame for color camera (including 0x81 and 0x85 packets). All packets are 1920 bytes except 0x85 packet - 24 bytes. Minus header it gives 307200 bytes of data.
Possible improvements to this algorithm could include monitoring packet_seq and timecode for sequential data packets. Obviously if the data is corrupt you have bigger problems but it might be nice. Need to figure out the magic.
After we have the data, for basic display in the QWidget and to generate proper RGB pixels:
- Reverse the endianness of the data. (0b10000000 becomes 0b00000001)
- Shove it into an RGB bitstream, since we're only dealing with monochrome data it's just duplicated three times for convienience.
As a note there's some bit manipulation going on to get it into a pixel value. Each pixel is 11 bits, which gives it 2047 possible values it looks like. Kind of nifty. You can see the bit-shifting going on inside the libkinect source, documented here for prosperity:
for (int p = 0, bit_offset = 0; p < 640*480; p++, bit_offset += 11) {
uint32_t pixel = 0; // value of pixel
pixel = *((uint32_t *)(data+(p*11/8)));
pixel >>= (p*11 % 8);
pixel &= 0x7ff;
uint8_t pix_low = (pixel & 0x00ff) >> 0;
uint8_t pix_high = (pixel & 0xff00) >> 8;
pix_low = reverse[pix_low];
pix_high = reverse[pix_high];
pixel = (pix_low << 8) | (pix_high);
pixel >>= 5;
// Image drops the 3 MSbs
rgb[3*p+0] = 255-pixel;
rgb[3*p+1] = 255-pixel;
rgb[3*p+2] = 255-pixel;
}
Looks fairly routine, but I'll be the first to admit bit-wise operations make my head hurt, doubly so for efficient bit-wise operations.
RGB Camera
The RGB Camera follows the same frame format as the depth camera, with a minor difference of cmd being 0x8x instead of 0x7x.
The frame output of the RGB camera is a 640x480 Bayer pattern, so the total frame size is 640*480=307200 bytes.
The Bayer pattern is layed out as: RG, GB.
Control EP
Control endpoint uses different header
struct control_hdr {
uint8_t magic[2]; //can be "RB" for Input and "GM" for Output (command)
uint16_t len; //data length in words
uint16_t cmd;
uint16_t cmd2_or_tag; //cmd and tag from GM packet should be matched in response RB packet
uint16_t data[];
};
Control point is most likely inits the IR projector and cameras
Commands remain unknown (yet)
NUI Audio
So far not much has been determined about the audio interface. It looks as if there might be bi-directional streams shooting back and forth, the XBox might be exporting some of it's own audio to the Kinect for some type of echo cancelation. Filtering out game noises certainly would be a clever thing to do.
Getting audio working might be challenging, but rewarding if built-in echo cancelation is present.