SC-Depth: Unsupervised Scale-consistent Depth Estimation (IJCV 2021 & NeurIPS 2019)

August 28, 2019 33 Comments

Notice: this post presents the NeurIPS version. Please see more information in our IJCV paper [pdf].

Abstract

Recent work has shown that CNN-based depth and ego-motion estimators can be learned using unlabelled monocular videos. However, the performance is limited by unidentified moving objects that violate the underlying static scene assumption in geometric image reconstruction. More significantly, due to lack of proper constraints, networks output scale-inconsistent results over different samples, i.e., the ego-motion network cannot provide full camera trajectories over a long video sequence because of the per-frame scale ambiguity. This paper tackles these challenges by proposing a geometry consistency loss for scale-consistent predictions and an induced self-discovered mask for handling moving objects and occlusions. Since we do not leverage multi-task learning like recent works, our framework is much simpler and more efficient. Comprehensive evaluation results demonstrate that our depth estimator achieves the state-of-the-art performance on the KITTI dataset. Moreover, we show that our ego-motion network is able to predict a globally scale-consistent camera trajectory for long video sequences, and the resulting visual odometry accuracy is competitive with the recent model that is trained using stereo videos. To the best of our knowledge, this is the first work to show that deep networks trained using unlabelled monocular videos can predict globally scale-consistent camera trajectories over a long video sequence.

Publication

Unsupervised Scale-consistent Depth Learning from Video, Jia-Wang Bian, Huangying Zhan, Naiyan Wang, Zhichao Li, Le Zhang, Chunhua Shen, Ming-Ming Cheng, Ian Reid, IJCV, 2021. [pdf | code] (Extended version of NeurIPS 2019)

@article{bian2021ijcv, 
  title={Unsupervised Scale-consistent Depth Learning from Video}, 
  author={Bian, Jia-Wang and Zhan, Huangying and Wang, Naiyan and Li, Zhichao and Zhang, Le and Shen, Chunhua and Cheng, Ming-Ming and Reid, Ian}, 
  journal= {International Journal of Computer Vision (IJCV)}, 
  year={2021} 
}

Contributions

  1. We propose a geometry consistency constraint to enforce the scale-consistency of depth and ego-motion networks, leading to globally scale-consistent results.
  2. We propose a self-discovered mask for detecting dynamics and occlusions by the aforementioned geometry consistency constraint. Compared with other approaches, our proposed approach does not require additional optical flow or semantic segmentation networks, which makes the learning framework simpler and more efficient.
  3. The proposed depth estimator achieves state-of-the-art performance on KITTI dataset, and the proposed ego-motion predictor shows competitive visual odometry results compared with the state-of-the-art model that is trained using stereo videos.

Proposed Framework

  1. LGC stands for the proposed geometry-consistency loss. It penalizes the inconsistency of depth predictions on consecutive frames, i.e., the difference between the predicted depth and the projected depth (from the other frame). By constraining this, the scale-consistency could be enforced. Besides, it regularizes the network and overcomes the overfitting issue. See the paper for details.
  2. M stands for the proposed self-discovered mask, which is derived from LGC. Specifically, it is a confidence map (how the predicted depth is consistent with the projected depth), and it is normalized to the range (0,1). We apply this as a weight mask during photometric loss calculation. The low-weight regions are detected dynamics and occlusions.
  3. Other parts are similar to SfMLearner (Zhou et al. [5]).

Visual Results of Depth and Mask

  1. Top to bottom: two consecutive images, estimated depth, proposed mask. White (black) stand for high (low) confidences.
  2. Note that only static and co-viewed regions by both images can provide reasonable supervisions, and other regions are noises in this geometry-based framework.
  3. The proposed mask, derived from geometry, can effectively detect good regions and remove noisy ones (i.e., moving objects and occlusions).

Depth Results on KITTI

  1. We use Eigen’s split for training and testing (standard solution).
  2. The methods trained on KITTI raw dataset are denoted by K. Models with pre-training on CityScapes are denoted by CS+K.
  3. (D) denotes depth supervision, (B) denotes binocular/stereo input pairs, (M) denotes monocular video clips. (J) denotes joint learning of multiple tasks. The best performance is highlighted as bold.

Visual Odometry Results

  1. All deep methods are trained on KITTI 00-08. ORB-SLAM (without loop closing) is compared as the strong baseline.
  2. Zhou et al. [5] use monocular videos for training. We align its scale of each frame to ground truth, because its scale is not consistent.
  3. Zhan et al. [16] use stereo videos for training, so no scale ambiguity.
  4. Our method uses monocular videos for training, but only aligns one global scale to the ground truth. The results are comparable and even better than [16].

Efficiency of Training

  1. We compare with CC [9] on a single 16GB Tesla V100 GPU. The time taken for each iteration consisting of forward and backward pass using a batch size of 4 is reported, where image resolution is 832 × 256.
  2. CC [9] needs train 3 parts iteratively, while we only need train 1 part once for 200K iterations. CC takes about 7 days for training, while our method takes 32 hours.

Selected Reference

  • [5] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. CVPR, 2017.
  • [9] Anurag Ranjan, Varun Jampani, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black. Competitive Collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. CVPR, 2019.
  • [16] Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. CVPR, 2018.

Reconstruction Demo

33 thoughts on “SC-Depth: Unsupervised Scale-consistent Depth Estimation (IJCV 2021 & NeurIPS 2019)”

  6. Dear author:

    I downloaded your pretrained model(depth), but I got an error when uncompressing it. Then I changed my account and used a new PC, it failed again. I guess that your uploaded models maybe have some problems. It would be a better idea if you could check the link or something. thank you very much.

  7. It does not need to be upcompressed. You just need pass its location (e.g., “~/Research/SC-Models/cs+k_depth.tar”) to the evaluation code.

  9. Hi, Thanks for sharing great work.
    May I ask you sharing full Pseudo RGB-D SLAM system code?
    It would be very grateful.

    Many thanks!

  10. You may need to implement it by yourself. You just need to save the depth prediction, and then feed it to ORB-SLAM2.

  11. On this site, you can discover a wide range virtual gambling platforms.
    Whether you’re looking for well-known titles new slot machines, there’s something for every player.
    All featured casinos fully reviewed to ensure security, so you can play with confidence.
    casino
    What’s more, the platform offers exclusive bonuses and deals to welcome beginners as well as regulars.
    With easy navigation, finding your favorite casino takes just moments, making it convenient.
    Be in the know about the latest additions by visiting frequently, as fresh options are added regularly.

  12. Attractive component to content. I simply stubled upon your website and in acceseion capital to claim that
    I get actually loved account your blog posts.
    Any way I will be subscribing for your augment or even I achievement yyou
    get right of entry to persistently fast.

  13. Here, you can discover a great variety of casino slots from top providers.
    Visitors can experience retro-style games as well as feature-packed games with stunning graphics and exciting features.
    Even if you’re new or a seasoned gamer, there’s always a slot to match your mood.
    money casino
    Each title are available round the clock and compatible with PCs and mobile devices alike.
    No download is required, so you can get started without hassle.
    The interface is user-friendly, making it simple to find your favorite slot.
    Register now, and dive into the thrill of casino games!

  19. [url=https://samoylovaoxana.ru/tag/ekskursii-v-dagestan-iz-moskvy/]экскурсии в Дагестан из Москвы[/url] или [url=https://samoylovaoxana.ru/tag/egipet/]Египет[/url]

    [url=https://samoylovaoxana.ru/tag/puteshestvie-po-rossii/]путешествие по России[/url]

    https://samoylovaoxana.ru/v-aby-dabi-otkryli-immersivnyi-park-s-krypneishim-akvariymom-v-mire/

    Ещё можно узнать: [url=http://yourdesires.ru/it/1837-kak-postavit-simvol-v-word-neskolko-sposobov.html]знак параграф в ворде[/url]

  20. Hello team!
    I came across a 113 awesome website that I think you should dive into.
    This resource is packed with a lot of useful information that you might find valuable.
    It has everything you could possibly need, so be sure to give it a visit!
    [url=https://nothing2hide.net/health-lifestyle/autumn-blues-of-the-body-how-to-protect-skin-nails-and-hair/]https://nothing2hide.net/health-lifestyle/autumn-blues-of-the-body-how-to-protect-skin-nails-and-hair/[/url]

  21. Some French players like Kingsley Coman, Randal Kolo Muani and Aurélien Tchouaméni had been racially abused
    on-line attributable to their performances by their own followers.

    In addition to that, you can evaluate historical tournaments courting back to 1998 to grasp previous winners and workforce
    performances at different stages of the FIFA World Cup by way of
    the standings available too. Additionally, you possibly can verify for
    type over numerous periods, customisable up to 30 games ought to that sample size be available for the World Cup.

    Brazil over Argentina? But why? But that ignores the fact that not solely do Brazil have a lot
    of improving to do in a brief space of time to achieve their potential however Argentina have just proved that in Julián Álvarez and Co., they’ve a agency reply to the publish-peak-Messi period.
    This web page will be updated with extra data and bets
    when the date nears nearer and lead up video games have been determined.
    Who will win the Soccer World Cup in 2026? In in the present day’s betting odds information, we’re diving
    deeper into the FIFA World Cup odds and soccer betting lines.
    African groups have by no means won a World Cup.

  22. Here, you can discover a wide range internet-based casino sites.
    Searching for classic games latest releases, you’ll find an option for every player.
    The listed platforms checked thoroughly to ensure security, so you can play with confidence.
    casino
    Additionally, the platform provides special rewards plus incentives for new players including long-term users.
    Thanks to user-friendly browsing, discovering a suitable site takes just moments, saving you time.
    Be in the know on recent updates with frequent visits, because updated platforms come on board often.

  23. The Aviator Game combines adventure with big wins.
    Jump into the cockpit and play through cloudy adventures for huge multipliers.
    With its retro-inspired design, the game evokes the spirit of aircraft legends.
    https://www.linkedin.com/posts/robin-kh-150138202_aviator-game-download-activity-7295792143506321408-81HD/
    Watch as the plane takes off – withdraw before it flies away to secure your rewards.
    Featuring instant gameplay and immersive background music, it’s a must-try for casual players.
    Whether you’re chasing wins, Aviator delivers endless thrills with every flight.

  25. 本站 提供 海量的 成人资源，满足 各类人群 的 喜好。
    无论您喜欢 哪种类型 的 影片，这里都 应有尽有。
    所有 内容 都经过 严格审核，确保 高清晰 的 观看体验。
    偷窥
    我们支持 各种终端 访问，包括 平板，随时随地 尽情观看。
    加入我们，探索 绝妙体验 的 私密乐趣。

  26. Уборка квартир в СПб! Больше времени на себя, а не на уборку! Профессиональный клининг. Цены от 1590 руб.. Закажите сейчас! Переходите https://uborka-kvartir24top.ru

  27. Hello friends!
    I came across a 113 valuable website that I think you should explore.
    This site is packed with a lot of useful information that you might find interesting.
    It has everything you could possibly need, so be sure to give it a visit!
    [url=https://gamingspell.com/how-the-inoffensive-vape-can-sneakily-destroy-health-more-quickly-than-traditional-smoking-cigarettes/]https://gamingspell.com/how-the-inoffensive-vape-can-sneakily-destroy-health-more-quickly-than-traditional-smoking-cigarettes/[/url]

  30. На нашей платформе интимные фото и ролики.
    Контент подходит тем, кто старше 18.
    У нас собраны разные стили и форматы.
    Платформа предлагает четкие фото.
    Dioxafetyl butyrate
    Вход разрешен после подтверждения возраста.
    Наслаждайтесь эксклюзивным контентом.

  31. Трендовые фасоны сезона 2025 года отличаются разнообразием.
    Актуальны кружевные рукава и корсеты из полупрозрачных тканей.
    Металлические оттенки придают образу роскоши.
    Греческий стиль с драпировкой возвращаются в моду.
    Минималистичные силуэты подчеркивают элегантность.
    Ищите вдохновение в новых коллекциях — оригинальность и комфорт сделают ваш образ идеальным!
    http://xn--12cg0dgd0cgkso9a9eg1b0dvhwf.com/index.php?topic=60677.new#new

  32. На этом сайте вы найдете подготовительные ресурсы для абитуриентов.
    Все школьные дисциплины в одном месте от математики до литературы.
    Готовьтесь к ЕГЭ и ОГЭ благодаря интерактивным заданиям.
    https://newslab.ru/news/1332715
    Демонстрационные варианты объяснят сложные моменты.
    Регистрация не требуется для удобства обучения.
    Применяйте на уроках и достигайте отличных результатов.

  33. Wow that was unusual. I just wrote an really long comment but after I clicked submit my
    comment didn’t show up. Grrrr… well I’m not writing all
    that over again. Regardless, just wanted to say excellent blog!

Leave a Reply

Your email address will not be published. Required fields are marked *