Stran je zaklenjena

Stran je zaklenjena za urejanje. Počakati je treba, da zaklep strani poteče.
Trenutno je zaklenjeno s strani: 216.73.216.110,10.0.1.7
Zaklep preteče ob: 2026/01/26 03:50 (13 min)
Stran je odprta z dovoljenji le za branje. Dovoljeno je ogledati si izvorno kodo strani, vsebine pa ni mogoče spreminjati. Za več podrobnosti stopite v stik s skrbnikom sistema.
FIXME **This page is not fully translated, yet. Please help completing the translation.**\\ //(remove this paragraph once the translation is finished)//

{{ :progetti:5p4k:ratcam_alpha.jpg?direct&180|}}
====== Ratcam v0 ======
> Ratcam allows you to control your RaspberryPi camera through a Telegram bot, and alerts you when movement is detected, by sending photos and videos.

The idea came to me when I found traces of mice in my house; rather than adopting drastic measures (and having to deal with the remains), I wanted to figure out from where they entered.

==== Project status ====
<row>
<col md="9">
Complete!! See the upgrades planned for [[ratcam v1]]

  - <badge>OK</badge> Minimal bot support
  - <badge>OK</badge> Sending photos and videos
  - <badge>OK</badge> Muxing H.264 into a MP4 file
     - <badge>OK</badge> Integration of the MP4 muxer as Picamera encoder
     - <badge>OK</badge> Pull request to Picamera 8-)
  - <badge>OK</badge> Motion detection
  - <badge>OK</badge> Integration of all functionalities
  - <badge>OK</badge> Authentication of Telegram chats
  - <badge>OK</badge> MP4 circular buffer
  - <badge>OK</badge> Parallelization of bot and monitor process
  - <badge>OK</badge> IR lighting setup
  - <badge>OK</badge> Camera calibration

--- //[[user:5p4k|Pietro Saccardi]] 2017/10/07 15:22//

</col>
<col md="3">
<panel type="default" title="Ratcam" no-body="true">
<list-group>
  * **Began:** Sept 4, 2017
  * **Ended:** Oct 7, 2017
  * **Status:** 100% complete
</list-group>
[[https://git.mittelab.org/proj/ratcam|Code]]

Upgrade: [[ratcam v1]]
</panel>
</col>
</row>

==== Functionalities ====
  * (Night) monitoring
  * Motion trigger
  * Automatic video recording and sending triggered by motion
  * Photo and video on demand
  * Telegram bot control

{{ :progetti:5p4k:ratcam_work_table.jpg?direct&300|}} ==== Materials ====
Hardware:
   * RaspberryPi (I use a model **2B**)
   * USB WiFi Antenna for Raspberry (or ethernet connection)
   * RaspberryPi Camera Module (I use the model **noIR v2**)
   * IR LED for lighting the scene and resistors: see [[#Night camera]]
Software:
  * The software is entirely in **Python 3**
  * You should have a working camera setup for the Raspberry, see [[#Installation]]

===== Implementation details =====
This section explains which software and libraries have been used.

=== Camera/bot interaction ===

The plan is to have the following interactions possible:
  * The user can request at any moment photo or video
  * We analyse the frames to detect motion, and when detected, a notification is sent.
  * The camera records on a circular buffer, in such a way that when motion is detected, we have already recorded the few seconds preceding the event.
  * We record as long as motion is detected, breaking the video every 30s or 1 minute, sending it through the Telegram bot
  * The user can request deactivation of the camera
  * When motion is detected, the camera sends a heat map picture of the motion.

{{ :progetti:5p4k:ratcam_telegram.png?direct&200|}}=== Telegram bot ===
The library used is [[https://github.com/python-telegram-bot/python-telegram-bot|python-telegram-bot]], which has an [[https://python-telegram-bot.readthedocs.io/en/stable/|excellent documentation]] and supports photos and videos. Their [[https://github.com/python-telegram-bot/python-telegram-bot/wiki|wiki]] contains tons of snippets and examples.

A Telegram bot was chosen to control the camera for the simplicity of use and the availability of several libraries. Using a bot lifts me from the responsibility of managing a network protocol, a server or a simple direct connection. Moreover enables cloud storage for the media and handles automatically all requests.

=== Audio/video management ===
The library used is called [[https://github.com/waveform80/picamera/|picamera]]. It's the //de-facto// standard to handle audio/video on the Raspberry Pi; it's [[http://picamera.readthedocs.io/en/release-1.12/index.html|well documented]] as well. Picamera provides some functionalities that are essential for this project:
  * Writing on any stream ([[https://picamera.readthedocs.io/en/release-1.13/recipes1.html#capturing-to-a-stream|guide]])
  * Low light image control ([[https://picamera.readthedocs.io/en/release-1.13/recipes1.html#capturing-in-low-light|guide]])
  * Native H.264 encoding ([[https://picamera.readthedocs.io/en/release-1.13/recipes1.html#recording-video-to-a-stream|guide]]). This is of the essence: an uncompressed video would be too big for the RaspberryPi computational capabilities, aside not being supported by Telegram. Picamera accesses directly the API of ''libmmal'', which returns a stream that is pre-encoded by the dedicated hardware on the RasperryPi.
  * Simultaneous capture and analysis ([[https://picamera.readthedocs.io/en/release-1.13/recipes2.html#rapid-capture-and-processing|guide]])
  * Extraction of the H.264 motion vector (see [[#H.264 and MP4]]) while encoding ([[https://picamera.readthedocs.io/en/release-1.13/recipes2.html#recording-motion-vector-data|guide]]). This means that motion estimate can be done on the dedicated hardware (and by a well tested and effective algorithm). Motion detection is literally a dozen lines of code.

=== Multiprocess ===
In order to analyse the image and simultaneously interact with the bot, it is necessary to run two parallel processes. Python does not support multithreading (or rather, its [[https://stackoverflow.com/q/1294382/1749822|Global Interpreter Lock]] forces one thread to be executed at a time), therefore we must make the two process communicate. The [[https://docs.python.org/3.6/library/multiprocessing.html|multiprocessing]] library made it easy: an object shared between the two processes signals the events (motion detected, no more motion, photo/video request, photo/video ready to send) and dispatches the path of the files to the Telegram bot.

{{ :progetti:5p4k:ratcam_console.png?direct&400|}}
=== Authentication and multiple chats ===
Technically, a Telegram bot is accessible by anyone; but not everyone should be able to peek at the camera. I implemented an authentication system by prompting for a password that is printed to the console. The authentication status is persisted on a JSON file, and messages coming from unauthorized chats are blocked already at bot filter level. The software handles thus multiple chats, but it's sufficient to upload the media once, and then forward it to the other chats.

=== Motion detection ===
The H.264 motion vector is a vector estimate (direction and speed) of the movement of the parts that compose the image. H.264 computes this as part of its own encoding mechanisms. The estimate is used to generate a prediction of the following frame; what will actually be stored is the part of the image that differs from the prediction. In the following table, I extracted the norm of the motion vector for each of the macroblocks of the original video, and represented it as grayscale. In the third column, I removed colors from the original video, and overlaid the motion vector on top of it.
<grid>
<col md="4"><thumbnail>{{ :progetti:5p4k:motionv_orig.mp4 |}}<caption>Original video</caption></thumbnail></col>
<col md="4"><thumbnail>{{ :progetti:5p4k:motionv.mp4 |}}<caption>Motion vector</caption></thumbnail></col>
<col md="4"><thumbnail>{{ :progetti:5p4k:motionv_overlay.mp4 |}}<caption>Overlay</caption></thumbnail></col>
</grid>

The H.264 motion vector is quite noisy, so I implemented a simple median filter in ''Pillow''; this improves remarkably the noise. These two images represent the motion vector (the squared norm of the vectors, normalized). The second picture clearly shows the effectiveness of the median filter.
<grid>
<col md="6"><thumbnail>{{:progetti:5p4k:normdata.png?nolink&400|}}<caption>Original</caption></thumbnail></col>
<col md="6"><thumbnail>{{:progetti:5p4k:normdata_med.png?nolink&400|}}<caption>Median filter</caption></thumbnail></col>
</grid>

The solution adopted:
  - Compute the norm of the motion vector
  - Apply the median filter to remove the noise
  - Sum the intensity of the frames over a time interval of few seconds
  - Let the older frames decay exponentially, in such a way that after that interval, the intensity falls below 1.

In this way we can catch slow motion too, because they accumulate over time in the same region, until they go beyond the activation threshold.

<grid>
<col><thumbnail>{{ :progetti:5p4k:ratcam_heat_map.jpg?direct&600 |}}<caption>Picture generated by Ratcam in which motion is represented via heat map.</caption></thumbnail></col>
</grid>

=== MP4 Muxing ===
The stream generated by Picamera is pyre H.264, not an MP4; Telegram accepts a well-formed MP4. We could convert it using a shell command like ''ffmpeg -f h264 -i pipe:0 -r <fps> -c:v copy -an -f mp4 -'' taking input from stdin and writing to stdout; however, in the beginning it didn't seem that complicated to mux an MP4 file starting from the H.264 stream directly in Python, instead of spawning a separate process and copying data back and forth. As a matter of fact, it was a bit more complicated (more details in [[#H.264 and MP4]]). Anyhow, I now have this extra functionality implemented in Python.

==== Night camera ====
The project works perfectly with a normal camera, in daylight. For night vision though, we need to take a bit of extra care.
  - The camera module for the Raspberry must be the **noIR** version.
  - We need to light up the scene (like a normal spotlight, just outside the human visible spectrum -- visible for the camera though)
<panel type="info" title="Watch out for the camera module name">
The camera module sensitive to IR is the [[https://www.raspberrypi.org/products/pi-noir-camera-v2/|noIR]]. The name is misleading, **no IR** does not mean that it's not sensitive to infrared light. It means instead that **it does //not// have a filter for infrared light**. Camera sensors by themselves **are** sensitive to IR; to avoid IR light contamination in the image, a filter that **removes** IR is added. The **noIR** camera module does **not** have such filter, and thus is sensitive to IR.
</panel>

[[http://www.conrad.com/ce/en/product/183338/LED-set-870-nm-925-nm-5-mm-Radial-lea;jsessionid=FDF2681AB87EBA6800EBFF1059B5218C.ASTPCEN27?ref=list|These]] are the LEDs I used; 1.35V, 100mA, at 870 and 925 nm respectively. At first sight, it doesn't seem that there is any meaningful sensitivity difference between 870 and 925 nm. Nonetheless, I'm using 870nm LEDs, just to get further away from the visible spectrum.

To power them up, I am using the [[https://pinout.xyz/pinout/pin2_5v_power|5V pins]] of the Raspberry (the [[https://pinout.xyz/pinout/pin1_3v3_power|3v3 pin]] does not support above 500mA, and we are being generous). These 5V pins are directly connected to the power source; I am using an adapter rated for 2A, so I should be on the safe side: the Raspberry Pi, even with a camera and actively recording, corrente; il [[http://raspi.tv/2016/raspberry-pi-zero-1-3-power-usage-with-camera|doesn't exceed]] the 500mA of current.

<grid>
<col md="6"><thumbnail>{{:progetti:5p4k:5led.jpg?direct&400|}}<caption>5 870nm LED powered by the RPi.</caption></thumbnail></col>
<col md="6"><thumbnail>{{:progetti:5p4k:5led_light_cone.jpg?direct&400|}}<caption>Light cone for 5 LED IR.</caption></thumbnail></col>
</grid>

I temporarily installed the LEDs on a cardboard ring, in such a way that they light the scene up in front of the camera, and sorted a bit out the cables. Lighting is quite good, setting the camera up with 800 ISO and night mode I can get quite clear images! Each LED is installed behind a 36Ω resistor, and they're connected in parallel.


<grid>
<col md="6"><thumbnail>{{ :progetti:5p4k:ratcam_living_room.jpg?direct&400 |}}<caption>Immagine presa con Ratcam</caption></thumbnail></col>
<col md="6"><thumbnail>{{ :progetti:5p4k:ratcam_resistors.jpg?direct&400 |}}<caption>L'installazione temporanea sulla breadboard.</caption></thumbnail></col>
</grid>


==== Installation ====
This section is about preparing the Raspberry.
  - I used [[https://www.raspberrypi.org/downloads/raspbian/|Raspbian Stretch Lite]], link to the [[https://www.raspberrypi.org/documentation/installation/installing-images/README.md|installation guide]].
  - Some guides to set WiFi and SSH up already before the first boot:
 [[https://raspberrypi.stackexchange.com/a/57023/72884|WiFi]] and [[https://caffinc.github.io/2016/12/raspberry-pi-3-headless/|SSH (section 4)]]
  - Update the software ''sudo apt-get update && sudo apt-get dist-upgrade''!
  - Install the camera module, link to the [[https://thepihut.com/blogs/raspberry-pi-tutorials/16021420-how-to-install-use-the-raspberry-pi-camera|guide I followed]] (remember to use ''raspi-config''!)
  - Prepare a healthy Python3 environment, clone the repository, install dependencies. I plan to use ''Pillow'' for image manipulation, and ''numpy'' for motion detection. Although they're not yet used in the code, you should install them. The following steps take into account the components that are needed for Pillow and numpy. References:
     - [[https://stackoverflow.com/a/33508237/1749822]]
     - [[https://www.raspberrypi.org/forums/viewtopic.php?f=91&t=79379#p571710]]
<file bash setup.sh>
# Environment
sudo apt-get install git virtualenv
# Minimal Pillow packages
sudo apt-get install python3-dev libjpeg-dev
# Required for cffi which is required for bcrypt
sudo apt-get install libffi-dev
git clone https://git.mittelab.org/proj/ratcam.git
cd ratcam
# Create virtual environment to keep system clean
virtualenv -p python3 venv
# Activate and load used packages
source venv/bin/activate
pip install -r requirements.txt
</file>


===== H.264 and MP4 =====
== (AKA: How to hurt oneself and come out unharmed) ==
Muxing a H.264 stream in an MP4 is not exactly trivial, but I think it's something that is really missing from Picamera. In this section I write down some difficulties I encountered and resources I used.

First of all, the official sources that deal with H.264 and MP4:
  * //ISO/IEC 14496-12// contains information on MP4 and the definition of all the main //atoms//.
  * //ISO/IEC 14496-15// contains information on AVC, especially the ''avc1'' e ''avcC'' //atom// which are necessary for correctly encoding the file.
  * //ISO/IEC 14496-10// contains all the details on H.264. I just needed a part of this, to understand how the stream is structured and how to distinguish the NALs.

Some essential contributions for a better understanding:
  * [[https://cardinalpeak.com/blog/the-h-264-sequence-parameter-set/|SPS/PPS info]]
  * [[https://stackoverflow.com/a/18024838/1749822|NAL marker question]]
  * [[https://stackoverflow.com/a/11869227/1749822|Question on how to encode profile and level]]
  * [[http://aviadr1.blogspot.de/2010/05/h264-extradata-partially-explained-for.html|Annex B info]]
  * [[http://blog.pearce.org.nz/2013/11/what-does-h264avc1-codecs-parameters.html|Profile encoding for H.264]]
  * [[http://gentlelogic.blogspot.de/2011/11/exploring-h264-part-2-h264-bitstream.html|Rough explanation of H.264]]
  * [[https://sourceforge.net/projects/h264bitstream/|h264bitstream]], tool to analyse H.264

Essentially, MP4 consists of a hierarchical tree of classes, called //atoms///, which specify all the properties of the file. //ISO/IEC 14496-12// defines all these properties and parts, and explains in detail how to encode them and which are mandatory. Atoms are stored in binary format with a 32bit field that contains the overall atom length, followed by a 4 byte string which specifies the type (it's human readable). Data follows.

We must rebuild and encode all atoms. For this there is a library (there's a library for everything {{Citation needed}}), [[https://github.com/beardypig/pymp4|pymp4]]. Their [[https://github.com/beardypig/pymp4/blob/master/src/pymp4/cli.py|CLI tool]] was essential to spot the differences between the Python generated file and the ffmpeg one (best way of checking correctness).

The data needed to rebuild all atoms is limited: resolution, framerate, each sample's size (in bytes), profile and level of the stream, and SPS and PPS parameters. With the exception of profile, level and parameters, all the other information is available already in Picamera.

Once built the atoms, I had one more problem: the H.264 stream as it is can't be inserted into an ''mdat'' atom. QuickTime on OSX cannot read it, nor Telegram can. A binary diff of the ffmpeg-muxed file and the python-muxed file shows that there is a difference, which at first looks like a [[https://github.com/waveform80/picamera/issues/439|padding error]]. Upon further analysis, it turns out it's a change that happens always in the first 4 bytes of each frame, which are always ''00 00 00 01''. This is the separator between NAL units, which contain the several frame types (I, P or B) and the image parameters.

Essentially, the Annex B encoding for H.264 states that such separator needs to be replaced, within an MP4 file, with the length of the NAL unit that follows. All the H.264 frames sent by ''libmmal'', with the exception of the SPS and PPS parameters, have a single NAL unit, thus it's easy to patch it (before closing the stream). Frames that contain only SPS or PPS data, have (at least) two NALs, therefore we need to parse them (an easy task, just splitting at the separator). This allows us also to extract directly SPS and PPS parameters, that have to be inserted into the ''avcC'' atom, without having to pre-compute them from the MMAL encoder settings.

All of this ended up into a very simple MP4 muxer for a single H.264 stream, written in Python, which could be a good addition to Picamera, to produce directly MP4 files, without using external tools.

===== Polemics and trivia =====

=== Telegram bot: privacy and security ===
Media sent through Telegram are stored in the cloud, therefore the privacy level one can obtain is no better than the one provided by normal Telegram chats (not those end-to-end encrypted). The security of the transfer is the same as performing an HTTPS request (the protocol on top of which Telegram APIs travel), taking in account possible weaknesses in the implementation of python-telegram-bot and Telegram's inner protocol, on the client side, MTProto. This project is born to catch a glimpse of a mouse, not to do video surveillance. Implementing a whole network stack to transfer the media in a secure way is a bit overkill, and raises even more security problems.

=== "But... MTProto is not safe" ===
That's not quite correct, [[https://security.stackexchange.com/a/57792|here]] there is a description of what MTProto does or does not do. The mouse in my house is going to take the risk. After all, the polemics on the effectiveness of a custom-made protocol is always open =) At least Telegram [[https://core.telegram.org/techfaqoffers a prize]] for who breaks the protocol.


=== Why not using something pre-made? ===
Because I won't learn anything if I do. I would have never read ISO 14496 and learned tons of interesting stuff on H.264 if I hadn't tried to mux MP4.