Start moving to ytdl-org

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,61 @@
+## Please follow the guide below
+
+- You will be asked some questions and requested to provide some information, please read them **carefully** and answer honestly
+- Put an `x` into all the boxes [ ] relevant to your *issue* (like this: `[x]`)
+- Use the *Preview* tab to see what your issue will actually look like
+
+---
+
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2019.03.09*. If it's not, read [this FAQ entry](https://github.com/ytdl-org/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2019.03.09**
+
+### Before submitting an *issue* make sure you have:
+- [ ] At least skimmed through the [README](https://github.com/ytdl-org/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/ytdl-org/youtube-dl#faq) and [BUGS](https://github.com/ytdl-org/youtube-dl#bugs) sections
+- [ ] [Searched](https://github.com/ytdl-org/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones
+- [ ] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser
+
+### What is the purpose of your *issue*?
+- [ ] Bug report (encountered problems with youtube-dl)
+- [ ] Site support request (request for adding support for a new site)
+- [ ] Feature request (request for a new functionality)
+- [ ] Question
+- [ ] Other
+
+---
+
+### The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your *issue*
+
+---
+
+### If the purpose of this *issue* is a *bug report*, *site support request* or you are not completely sure provide the full verbose output as follows:
+
+Add the `-v` flag to **your command line** you run youtube-dl with (`youtube-dl -v <your command line>`), copy the **whole** output and insert it here. It should look similar to one below (replace it with **your** log inserted between triple ```):
+
+```
+[debug] System config: []
+[debug] User config: []
+[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
+[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
+[debug] youtube-dl version 2019.03.09
+[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
+[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
+[debug] Proxy map: {}
+...
+<end of log>
+```
+
+---
+
+### If the purpose of this *issue* is a *site support request* please provide all kinds of example URLs support for which should be included (replace following example URLs by **yours**):
+- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
+- Single video: https://youtu.be/BaW_jenozKc
+- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
+
+Note that **youtube-dl does not support sites dedicated to [copyright infringement](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
+
+---
+
+### Description of your *issue*, suggested solution and other information
+
+Explanation of your *issue* in arbitrary form goes here. Please make sure the [description is worded well enough to be understood](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient). Provide as much context and examples as possible.
+If work on your *issue* requires account credentials please provide them or explain how one can obtain them.

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -1,63 +0,0 @@
----
-name: Broken site support
-about: Report broken or misfunctioning site
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.06.21. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
-- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running youtube-dl version **2019.06.21**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar issues including closed ones
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of youtube-dl that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version 2019.06.21
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-```
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -1,54 +0,0 @@
----
-name: Site support request
-about: Request support for a new site
-title: ''
-labels: 'site-support-request'
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.06.21. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://yt-dl.org/copyright-infringement. youtube-dl does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running youtube-dl version **2019.06.21**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that none of provided URLs violate any copyrights
-- [ ] I've searched the bugtracker for similar site support requests including closed ones
-
-
-## Example URLs
-
-<!--
-Provide all kinds of example URLs support for which should be included. Replace following example URLs by yours.
--->
-
-- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
-- Single video: https://youtu.be/BaW_jenozKc
-- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
-
-
-## Description
-
-<!--
-Provide any additional information.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -1,37 +0,0 @@
----
-name: Site feature request
-about: Request a new functionality for a site
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.06.21. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running youtube-dl version **2019.06.21**
-- [ ] I've searched the bugtracker for similar site feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your site feature request in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -1,65 +0,0 @@
----
-name: Bug report
-about: Report a bug unrelated to any particular site or extractor
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.06.21. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
-- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Read bugs section in FAQ: http://yt-dl.org/reporting
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running youtube-dl version **2019.06.21**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar bug reports including closed ones
-- [ ] I've read bugs section in FAQ
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of youtube-dl that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version 2019.06.21
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-```
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -1,38 +0,0 @@
----
-name: Feature request
-about: Request a new functionality unrelated to any particular site or extractor
-title: ''
-labels: 'request'
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.06.21. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running youtube-dl version **2019.06.21**
-- [ ] I've searched the bugtracker for similar feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/6_question.md

@@ -1,38 +0,0 @@
----
-name: Ask question
-about: Ask youtube-dl related question
-title: ''
-labels: 'question'
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- Look through the README (http://yt-dl.org/readme) and FAQ (http://yt-dl.org/faq) for similar questions
-- Search the bugtracker for similar questions: http://yt-dl.org/search-issues
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm asking a question
-- [ ] I've looked through the README and FAQ for similar questions
-- [ ] I've searched the bugtracker for similar questions including closed ones
-
-
-## Question
-
-<!--
-Ask your question in an arbitrary form. Please make sure it's worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient.
--->
-
-WRITE QUESTION HERE

.github/ISSUE_TEMPLATE_tmpl.md

@@ -0,0 +1,61 @@
+## Please follow the guide below
+
+- You will be asked some questions and requested to provide some information, please read them **carefully** and answer honestly
+- Put an `x` into all the boxes [ ] relevant to your *issue* (like this: `[x]`)
+- Use the *Preview* tab to see what your issue will actually look like
+
+---
+
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *%(version)s*. If it's not, read [this FAQ entry](https://github.com/ytdl-org/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **%(version)s**
+
+### Before submitting an *issue* make sure you have:
+- [ ] At least skimmed through the [README](https://github.com/ytdl-org/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/ytdl-org/youtube-dl#faq) and [BUGS](https://github.com/ytdl-org/youtube-dl#bugs) sections
+- [ ] [Searched](https://github.com/ytdl-org/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones
+- [ ] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser
+
+### What is the purpose of your *issue*?
+- [ ] Bug report (encountered problems with youtube-dl)
+- [ ] Site support request (request for adding support for a new site)
+- [ ] Feature request (request for a new functionality)
+- [ ] Question
+- [ ] Other
+
+---
+
+### The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your *issue*
+
+---
+
+### If the purpose of this *issue* is a *bug report*, *site support request* or you are not completely sure provide the full verbose output as follows:
+
+Add the `-v` flag to **your command line** you run youtube-dl with (`youtube-dl -v <your command line>`), copy the **whole** output and insert it here. It should look similar to one below (replace it with **your** log inserted between triple ```):
+
+```
+[debug] System config: []
+[debug] User config: []
+[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
+[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
+[debug] youtube-dl version %(version)s
+[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
+[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
+[debug] Proxy map: {}
+...
+<end of log>
+```
+
+---
+
+### If the purpose of this *issue* is a *site support request* please provide all kinds of example URLs support for which should be included (replace following example URLs by **yours**):
+- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
+- Single video: https://youtu.be/BaW_jenozKc
+- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
+
+Note that **youtube-dl does not support sites dedicated to [copyright infringement](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
+
+---
+
+### Description of your *issue*, suggested solution and other information
+
+Explanation of your *issue* in arbitrary form goes here. Please make sure the [description is worded well enough to be understood](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient). Provide as much context and examples as possible.
+If work on your *issue* requires account credentials please provide them or explain how one can obtain them.

.github/ISSUE_TEMPLATE_tmpl/1_broken_site.md

@@ -1,63 +0,0 @@
----
-name: Broken site support
-about: Report broken or misfunctioning site
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is %(version)s. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
-- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running youtube-dl version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar issues including closed ones
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of youtube-dl that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version %(version)s
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-```
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md

@@ -1,54 +0,0 @@
----
-name: Site support request
-about: Request support for a new site
-title: ''
-labels: 'site-support-request'
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is %(version)s. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://yt-dl.org/copyright-infringement. youtube-dl does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running youtube-dl version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that none of provided URLs violate any copyrights
-- [ ] I've searched the bugtracker for similar site support requests including closed ones
-
-
-## Example URLs
-
-<!--
-Provide all kinds of example URLs support for which should be included. Replace following example URLs by yours.
--->
-
-- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
-- Single video: https://youtu.be/BaW_jenozKc
-- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
-
-
-## Description
-
-<!--
-Provide any additional information.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md

@@ -1,37 +0,0 @@
----
-name: Site feature request
-about: Request a new functionality for a site
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is %(version)s. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running youtube-dl version **%(version)s**
-- [ ] I've searched the bugtracker for similar site feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your site feature request in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/4_bug_report.md

@@ -1,65 +0,0 @@
----
-name: Bug report
-about: Report a bug unrelated to any particular site or extractor
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is %(version)s. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
-- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Read bugs section in FAQ: http://yt-dl.org/reporting
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running youtube-dl version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar bug reports including closed ones
-- [ ] I've read bugs section in FAQ
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of youtube-dl that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version %(version)s
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-```
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/5_feature_request.md

@@ -1,38 +0,0 @@
----
-name: Feature request
-about: Request a new functionality unrelated to any particular site or extractor
-title: ''
-labels: 'request'
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is %(version)s. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running youtube-dl version **%(version)s**
-- [ ] I've searched the bugtracker for similar feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.travis.yml

@@ -9,7 +9,6 @@ python:
   - "3.6"
   - "pypy"
   - "pypy3"
-dist: trusty
 env:
   - YTDL_TEST_SET=core
   - YTDL_TEST_SET=download

ChangeLog

@@ -1,246 +1,3 @@
-version 2019.06.21
-
-Core
-* [utils] Restrict parse_codecs and add theora as known vcodec (#21381)
-
-Extractors
-* [youtube] Update signature function patterns (#21469, #21476)
-* [youtube] Make --write-annotations non fatal (#21452)
-+ [sixplay] Add support for rtlmost.hu (#21405)
-* [youtube] Hardcode codec metadata for av01 video only formats (#21381)
-* [toutv] Update client key (#21370)
-+ [biqle] Add support for new embed domain
-* [cbs] Improve DRM protected videos detection (#21339)
-
-
-version 2019.06.08
-
-Core
-* [downloader/common] Improve rate limit (#21301)
-* [utils] Improve strip_or_none
-* [extractor/common] Strip src attribute for HTML5 entries code (#18485,
-  #21169)
-
-Extractors
-* [ted] Fix playlist extraction (#20844, #21032)
-* [vlive:playlist] Fix video extraction when no playlist is found (#20590)
-+ [vlive] Add CH+ support (#16887, #21209)
-+ [openload] Add support for oload.website (#21329)
-+ [tvnow] Extract HD formats (#21201)
-+ [redbulltv] Add support for rrn:content URLs (#21297)
-* [youtube] Fix average rating extraction (#21304)
-+ [bitchute] Extract HTML5 formats (#21306)
-* [cbsnews] Fix extraction (#9659, #15397)
-* [vvvvid] Relax URL regular expression (#21299)
-+ [prosiebensat1] Add support for new API (#21272)
-+ [vrv] Extract adaptive_hls formats (#21243)
-* [viki] Switch to HTTPS (#21001)
-* [LiveLeak] Check if the original videos exist (#21206, #21208)
-* [rtp] Fix extraction (#15099)
-* [youtube] Improve DRM protected videos detection (#1774)
-+ [srgssrplay] Add support for popupvideoplayer URLs (#21155)
-+ [24video] Add support for porno.24video.net (#21194)
-+ [24video] Add support for 24video.site (#21193)
-- [pornflip] Remove extractor
-- [criterion] Remove extractor (#21195)
-* [pornhub] Use HTTPS (#21061)
-* [bitchute] Fix uploader extraction (#21076)
-* [streamcloud] Reduce waiting time to 6 seconds (#21092)
-- [novamov] Remove extractors (#21077)
-+ [openload] Add support for oload.press (#21135)
-* [vivo] Fix extraction (#18906, #19217)
-
-
-version 2019.05.20
-
-Core
-+ [extractor/common] Move workaround for applying first Set-Cookie header
-  into a separate _apply_first_set_cookie_header method
-
-Extractors
-* [safari] Fix authentication (#21090)
-* [vk] Use _apply_first_set_cookie_header
-* [vrt] Fix extraction (#20527)
-+ [canvas] Add support for vrtnieuws and sporza site ids and extract
-  AES HLS formats
-+ [vrv] Extract captions (#19238)
-* [tele5] Improve video id extraction
-* [tele5] Relax URL regular expression (#21020, #21063)
-* [svtplay] Update API URL (#21075)
-+ [yahoo:gyao] Add X-User-Agent header to dam proxy requests (#21071)
-
-
-version 2019.05.11
-
-Core
-* [utils] Transliterate "þ" as "th" (#20897)
-
-Extractors
-+ [cloudflarestream] Add support for videodelivery.net (#21049)
-+ [byutv] Add support for DVR videos (#20574, #20676)
-+ [gfycat] Add support for URLs with tags (#20696, #20731)
-+ [openload] Add support for verystream.com (#20701, #20967)
-* [youtube] Use sp field value for signature field name (#18841, #18927,
-  #21028)
-+ [yahoo:gyao] Extend URL regular expression (#21008)
-* [youtube] Fix channel id extraction (#20982, #21003)
-+ [sky] Add support for news.sky.com (#13055)
-+ [youtube:entrylistbase] Retry on 5xx HTTP errors (#20965)
-+ [francetvinfo] Extend video id extraction (#20619, #20740)
-* [4tube] Update token hosts (#20918)
-* [hotstar] Move to API v2 (#20931)
-* [fox] Fix API error handling under python 2 (#20925)
-+ [redbulltv] Extend URL regular expression (#20922)
-
-
-version 2019.04.30
-
-Extractors
-* [openload] Use real Chrome versions (#20902)
-- [youtube] Remove info el for get_video_info request
-* [youtube] Improve extraction robustness
-- [dramafever] Remove extractor (#20868)
-* [adn] Fix subtitle extraction (#12724)
-+ [ccc] Extract creator (#20355)
-+ [ccc:playlist] Add support for media.ccc.de playlists (#14601, #20355)
-+ [sverigesradio] Add support for sverigesradio.se (#18635)
-+ [cinemax] Add support for cinemax.com
-* [sixplay] Try extracting non-DRM protected manifests (#20849)
-+ [youtube] Extract Youtube Music Auto-generated metadata (#20599, #20742)
-- [wrzuta] Remove extractor (#20684, #20801)
-* [twitch] Prefer source format (#20850)
-+ [twitcasting] Add support for private videos (#20843)
-* [reddit] Validate thumbnail URL (#20030)
-* [yandexmusic] Fix track URL extraction (#20820)
-
-
-version 2019.04.24
-
-Extractors
-* [youtube] Fix extraction (#20758, #20759, #20761, #20762, #20764, #20766,
-  #20767, #20769, #20771, #20768, #20770)
-* [toutv] Fix extraction and extract series info (#20757)
-+ [vrv] Add support for movie listings (#19229)
-+ [youtube] Print error when no data is available (#20737)
-+ [soundcloud] Add support for new rendition and improve extraction (#20699)
-+ [ooyala] Add support for geo verification proxy
-+ [nrl] Add support for nrl.com (#15991)
-+ [vimeo] Extract live archive source format (#19144)
-+ [vimeo] Add support for live streams and improve info extraction (#19144)
-+ [ntvcojp] Add support for cu.ntv.co.jp
-+ [nhk] Extract RTMPT format
-+ [nhk] Add support for audio URLs
-+ [udemy] Add another course id extraction pattern (#20491)
-+ [openload] Add support for oload.services (#20691)
-+ [openload] Add support for openloed.co (#20691, #20693)
-* [bravotv] Fix extraction (#19213)
-
-
-version 2019.04.17
-
-Extractors
-* [openload] Randomize User-Agent (closes #20688)
-+ [openload] Add support for oladblock domains (#20471)
-* [adn] Fix subtitle extraction (#12724)
-+ [aol] Add support for localized websites
-+ [yahoo] Add support GYAO episode URLs
-+ [yahoo] Add support for streaming.yahoo.co.jp (#5811, #7098)
-+ [yahoo] Add support for gyao.yahoo.co.jp
-* [aenetworks] Fix history topic extraction and extract more formats
-+ [cbs] Extract smpte and vtt subtitles
-+ [streamango] Add support for streamcherry.com (#20592)
-+ [yourporn] Add support for sxyprn.com (#20646)
-* [mgtv] Fix extraction (#20650)
-* [linkedin:learning] Use urljoin for form action URL (#20431)
-+ [gdc] Add support for kaltura embeds (#20575)
-* [dispeak] Improve mp4 bitrate extraction
-* [kaltura] Sanitize embed URLs
-* [jwplatfom] Do not match manifest URLs (#20596)
-* [aol] Restrict URL regular expression and improve format extraction
-+ [tiktok] Add support for new URL schema (#20573)
-+ [stv:player] Add support for player.stv.tv (#20586)
-
-
-version 2019.04.07
-
-Core
-+ [downloader/external] Pass rtmp_conn to ffmpeg
-
-Extractors
-+ [ruutu] Add support for audio podcasts (#20473, #20545)
-+ [xvideos] Extract all thumbnails (#20432)
-+ [platzi] Add support for platzi.com (#20562)
-* [dvtv] Fix extraction (#18514, #19174)
-+ [vrv] Add basic support for individual movie links (#19229)
-+ [bfi:player] Add support for player.bfi.org.uk (#19235)
-* [hbo] Fix extraction and extract subtitles (#14629, #13709)
-* [youtube] Extract srv[1-3] subtitle formats (#20566)
-* [adultswim] Fix extraction (#18025)
-* [teamcoco] Fix extraction and add suport for subdomains (#17099, #20339)
-* [adn] Fix subtitle compatibility with ffmpeg
-* [adn] Fix extraction and add support for positioning styles (#20549)
-* [vk] Use unique video id (#17848)
-* [newstube] Fix extraction
-* [rtl2] Actualize extraction
-+ [adobeconnect] Add support for adobeconnect.com (#20283)
-+ [gaia] Add support for authentication (#14605)
-+ [mediasite] Add support for dashed ids and named catalogs (#20531)
-
-
-version 2019.04.01
-
-Core
-* [utils] Improve int_or_none and float_or_none (#20403)
-* Check for valid --min-sleep-interval when --max-sleep-interval is specified
-  (#20435)
-
-Extractors
-+ [weibo] Extend URL regular expression (#20496)
-+ [xhamster] Add support for xhamster.one (#20508)
-+ [mediasite] Add support for catalogs (#20507)
-+ [teamtreehouse] Add support for teamtreehouse.com (#9836)
-+ [ina] Add support for audio URLs
-* [ina] Improve extraction
-* [cwtv] Fix episode number extraction (#20461)
-* [npo] Improve DRM detection
-+ [pornhub] Add support for DASH formats (#20403)
-* [svtplay] Update API endpoint (#20430)
-
-
-version 2019.03.18
-
-Core
-* [extractor/common] Improve HTML5 entries extraction
-+ [utils] Introduce parse_bitrate
-* [update] Hide update URLs behind redirect
-* [extractor/common] Fix url meta field for unfragmented DASH formats (#20346)
-
-Extractors
-+ [yandexvideo] Add extractor
-* [openload] Improve embed detection
-+ [corus] Add support for bigbrothercanada.ca (#20357)
-+ [orf:radio] Extract series (#20012)
-+ [cbc:watch] Add support for gem.cbc.ca (#20251, #20359)
-- [anysex] Remove extractor (#19279)
-+ [ciscolive] Add support for new URL schema (#20320, #20351)
-+ [youtube] Add support for invidiou.sh (#20309)
-- [anitube] Remove extractor (#20334)
-- [ruleporn] Remove extractor (#15344, #20324)
-* [npr] Fix extraction (#10793, #13440)
-* [biqle] Fix extraction (#11471, #15313)
-* [viddler] Modernize
-* [moevideo] Fix extraction
-* [primesharetv] Remove extractor
-* [hypem] Modernize and extract more metadata (#15320)
-* [veoh] Fix extraction
-* [escapist] Modernize
-- [videomega] Remove extractor (#10108)
-+ [beeg] Add support for beeg.porn (#20306)
-* [vimeo:review] Improve config url extraction and extract original format
-  (#20305)
-* [fox] Detect geo restriction and authentication errors (#20208)
-
-
 version 2019.03.09
 
 Core

Makefile

@@ -1,7 +1,7 @@
 all: youtube-dl README.md CONTRIBUTING.md README.txt youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish supportedsites
 
 clean:
-	rm -rf youtube-dl.1.temp.md youtube-dl.1 youtube-dl.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dl.tar.gz youtube-dl.zsh youtube-dl.fish youtube_dl/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png CONTRIBUTING.md.tmp youtube-dl youtube-dl.exe
+	rm -rf youtube-dl.1.temp.md youtube-dl.1 youtube-dl.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dl.tar.gz youtube-dl.zsh youtube-dl.fish youtube_dl/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png CONTRIBUTING.md.tmp ISSUE_TEMPLATE.md.tmp youtube-dl youtube-dl.exe
 	find . -name "*.pyc" -delete
 	find . -name "*.class" -delete
 
@@ -78,12 +78,8 @@ README.md: youtube_dl/*.py youtube_dl/*/*.py
 CONTRIBUTING.md: README.md
 	$(PYTHON) devscripts/make_contributing.py README.md CONTRIBUTING.md
 
-issuetemplates: devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.md .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md .github/ISSUE_TEMPLATE_tmpl/4_bug_report.md .github/ISSUE_TEMPLATE_tmpl/5_feature_request.md youtube_dl/version.py
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.md .github/ISSUE_TEMPLATE/1_broken_site.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md .github/ISSUE_TEMPLATE/2_site_support_request.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md .github/ISSUE_TEMPLATE/3_site_feature_request.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/4_bug_report.md .github/ISSUE_TEMPLATE/4_bug_report.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/5_feature_request.md .github/ISSUE_TEMPLATE/5_feature_request.md
+.github/ISSUE_TEMPLATE.md: devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl.md  youtube_dl/version.py
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl.md .github/ISSUE_TEMPLATE.md
 
 supportedsites:
 	$(PYTHON) devscripts/make_supportedsites.py docs/supportedsites.md

README.md

@@ -642,7 +642,6 @@ The simplest case is requesting a specific format, for example with `-f 22` you
 You can also use a file extension (currently `3gp`, `aac`, `flv`, `m4a`, `mp3`, `mp4`, `ogg`, `wav`, `webm` are supported) to download the best quality format of a particular file extension served as a single file, e.g. `-f webm` will download the best quality format with the `webm` extension served as a single file.
 
 You can also use special names to select particular edge case formats:
-
  - `best`: Select the best quality format represented by a single file with video and audio.
  - `worst`: Select the worst quality format represented by a single file with video and audio.
  - `bestvideo`: Select the best quality video-only format (e.g. DASH video). May not be available.
@@ -659,7 +658,6 @@ If you want to download several formats of the same video use a comma as a separ
 You can also filter the video formats by putting a condition in brackets, as in `-f "best[height=720]"` (or `-f "[filesize>10M]"`).
 
 The following numeric meta fields can be used with comparisons `<`, `<=`, `>`, `>=`, `=` (equals), `!=` (not equals):
-
  - `filesize`: The number of bytes, if known in advance
  - `width`: Width of the video, if known
  - `height`: Height of the video, if known
@@ -670,7 +668,6 @@ The following numeric meta fields can be used with comparisons `<`, `<=`, `>`, `
  - `fps`: Frame rate
 
 Also filtering work for comparisons `=` (equals), `^=` (starts with), `$=` (ends with), `*=` (contains) and following string meta fields:
-
  - `ext`: File extension
  - `acodec`: Name of the audio codec in use
  - `vcodec`: Name of the video codec in use
@@ -700,7 +697,7 @@ Note that on Windows you may need to use double quotes instead of single.
 # Download best mp4 format available or any other best if no mp4 available
 $ youtube-dl -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best'
 
-# Download best format available but no better than 480p
+# Download best format available but not better that 480p
 $ youtube-dl -f 'bestvideo[height<=480]+bestaudio/best[height<=480]'
 
 # Download best video only format but no bigger than 50 MB

devscripts/check-porn.py

@@ -45,12 +45,12 @@ for test in gettestcases():
 
         RESULT = ('.' + domain + '\n' in LIST or '\n' + domain + '\n' in LIST)
 
-    if RESULT and ('info_dict' not in test or 'age_limit' not in test['info_dict']
-                   or test['info_dict']['age_limit'] != 18):
+    if RESULT and ('info_dict' not in test or 'age_limit' not in test['info_dict'] or
+                   test['info_dict']['age_limit'] != 18):
         print('\nPotential missing age_limit check: {0}'.format(test['name']))
 
-    elif not RESULT and ('info_dict' in test and 'age_limit' in test['info_dict']
-                         and test['info_dict']['age_limit'] == 18):
+    elif not RESULT and ('info_dict' in test and 'age_limit' in test['info_dict'] and
+                         test['info_dict']['age_limit'] == 18):
         print('\nPotential false negative: {0}'.format(test['name']))
 
     else:

devscripts/release.sh

@@ -78,8 +78,8 @@ sed -i "s/__version__ = '.*'/__version__ = '$version'/" youtube_dl/version.py
 sed -i "s/<unreleased>/$version/" ChangeLog
 
 /bin/echo -e "\n### Committing documentation, templates and youtube_dl/version.py..."
-make README.md CONTRIBUTING.md issuetemplates supportedsites
-git add README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE/1_broken_site.md .github/ISSUE_TEMPLATE/2_site_support_request.md .github/ISSUE_TEMPLATE/3_site_feature_request.md .github/ISSUE_TEMPLATE/4_bug_report.md .github/ISSUE_TEMPLATE/5_feature_request.md .github/ISSUE_TEMPLATE/6_question.md docs/supportedsites.md youtube_dl/version.py ChangeLog
+make README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE.md supportedsites
+git add README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE.md docs/supportedsites.md youtube_dl/version.py ChangeLog
 git commit $gpg_sign_commits -m "release $version"
 
 /bin/echo -e "\n### Now tagging, signing and pushing..."

docs/supportedsites.md

@@ -28,7 +28,6 @@
  - **acast:channel**
  - **AddAnime**
  - **ADN**: Anime Digital Network
- - **AdobeConnect**
  - **AdobeTV**
  - **AdobeTVChannel**
  - **AdobeTVShow**
@@ -45,8 +44,9 @@
  - **AmericasTestKitchen**
  - **anderetijden**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
  - **AnimeOnDemand**
+ - **anitube.se**
  - **Anvato**
- - **aol.com**
+ - **AnySex**
  - **APA**
  - **Aparat**
  - **AppleConnect**
@@ -78,6 +78,7 @@
  - **AudioBoom**
  - **audiomack**
  - **audiomack:album**
+ - **auroravid**: AuroraVid
  - **AWAAN**
  - **awaan:live**
  - **awaan:season**
@@ -102,7 +103,6 @@
  - **Bellator**
  - **BellMedia**
  - **Bet**
- - **bfi:player**
  - **Bigflix**
  - **Bild**: Bild.de
  - **BiliBili**
@@ -149,7 +149,6 @@
  - **CBSInteractive**
  - **CBSLocal**
  - **cbsnews**: CBS News
- - **cbsnews:embed**
  - **cbsnews:livevideo**: CBS News Live Videos
  - **CBSSports**
  - **CCMA**
@@ -164,7 +163,6 @@
  - **chirbit**
  - **chirbit:profile**
  - **Cinchcast**
- - **Cinemax**
  - **CiscoLiveSearch**
  - **CiscoLiveSession**
  - **CJSW**
@@ -174,6 +172,7 @@
  - **Clipsyndicate**
  - **CloserToTruth**
  - **CloudflareStream**
+ - **cloudtime**: CloudTime
  - **Cloudy**
  - **Clubic**
  - **Clyp**
@@ -193,6 +192,7 @@
  - **Coub**
  - **Cracked**
  - **Crackle**
+ - **Criterion**
  - **CrooksAndLiars**
  - **crunchyroll**
  - **crunchyroll:playlist**
@@ -200,7 +200,6 @@
  - **CSpan**: C-SPAN
  - **CtsNews**: 華視新聞
  - **CTVNews**
- - **cu.ntv.co.jp**: Nippon Television Network
  - **Culturebox**
  - **CultureUnplugged**
  - **curiositystream**
@@ -236,6 +235,8 @@
  - **DouyuTV**: 斗鱼
  - **DPlay**
  - **DPlayIt**
+ - **dramafever**
+ - **dramafever:series**
  - **DRBonanza**
  - **Dropbox**
  - **DrTuber**
@@ -346,6 +347,7 @@
  - **Groupon**
  - **Hark**
  - **hbo**
+ - **hbo:episode**
  - **HearThisAt**
  - **Heise**
  - **HellPorno**
@@ -485,12 +487,9 @@
  - **MatchTV**
  - **MDR**: MDR.DE and KiKA
  - **media.ccc.de**
- - **media.ccc.de:lists**
  - **Medialaan**
  - **Mediaset**
  - **Mediasite**
- - **MediasiteCatalog**
- - **MediasiteNamedCatalog**
  - **Medici**
  - **megaphone.fm**: megaphone.fm embedded players
  - **Meipai**: 美拍
@@ -607,6 +606,7 @@
  - **nowness**
  - **nowness:playlist**
  - **nowness:series**
+ - **nowvideo**: NowVideo
  - **Noz**
  - **npo**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
  - **npo.nl:live**
@@ -622,7 +622,6 @@
  - **NRKTVEpisodes**
  - **NRKTVSeason**
  - **NRKTVSeries**
- - **NRLTV**
  - **ntv.ru**
  - **Nuvid**
  - **NYTimes**
@@ -632,6 +631,7 @@
  - **OdaTV**
  - **Odnoklassniki**
  - **OktoberfestTV**
+ - **on.aol.com**
  - **OnDemandKorea**
  - **onet.pl**
  - **onet.tv**
@@ -672,8 +672,6 @@
  - **Piksel**
  - **Pinkbike**
  - **Pladform**
- - **Platzi**
- - **PlatziCourse**
  - **play.fm**
  - **PlayPlusTV**
  - **PlaysTV**
@@ -690,6 +688,7 @@
  - **PopcornTV**
  - **PornCom**
  - **PornerBros**
+ - **PornFlip**
  - **PornHd**
  - **PornHub**: PornHub and Thumbzilla
  - **PornHubPlaylist**
@@ -699,6 +698,7 @@
  - **PornoXO**
  - **PornTube**
  - **PressTV**
+ - **PrimeShareTV**
  - **PromptFile**
  - **prosiebensat1**: ProSiebenSat.1 Digital
  - **puhutv**
@@ -718,7 +718,7 @@
  - **radio.de**
  - **radiobremen**
  - **radiocanada**
- - **radiocanada:audiovideo**
+ - **RadioCanadaAudioVideo**
  - **radiofrance**
  - **RadioJavan**
  - **Rai**
@@ -730,7 +730,6 @@
  - **RBMARadio**
  - **RDS**: RDS.ca
  - **RedBullTV**
- - **RedBullTVRrnContent**
  - **Reddit**
  - **RedditR**
  - **RedTube**
@@ -766,6 +765,7 @@
  - **RTVS**
  - **Rudo**
  - **RUHD**
+ - **RulePorn**
  - **rutube**: Rutube videos
  - **rutube:channel**: Rutube channels
  - **rutube:embed**: Rutube embedded videos
@@ -802,7 +802,6 @@
  - **ShowRoomLive**
  - **Sina**
  - **SkylineWebcams**
- - **SkyNews**
  - **skynewsarabia:article**
  - **skynewsarabia:video**
  - **SkySports**
@@ -853,10 +852,7 @@
  - **StreamCZ**
  - **StreetVoice**
  - **StretchInternet**
- - **stv:player**
  - **SunPorno**
- - **sverigesradio:episode**
- - **sverigesradio:publication**
  - **SVT**
  - **SVTPage**
  - **SVTPlay**: SVT Play and Öppet arkiv
@@ -877,7 +873,6 @@
  - **teachertube:user:collection**: teachertube.com user and collection videos
  - **TeachingChannel**
  - **Teamcoco**
- - **TeamTreeHouse**
  - **TechTalks**
  - **techtv.mit.edu**
  - **ted**
@@ -997,7 +992,6 @@
  - **Vbox7**
  - **VeeHD**
  - **Veoh**
- - **verystream**
  - **Vessel**
  - **Vesti**: Вести.Ru
  - **Vevo**
@@ -1016,11 +1010,13 @@
  - **video.mit.edu**
  - **VideoDetective**
  - **videofy.me**
+ - **VideoMega**
  - **videomore**
  - **videomore:season**
  - **videomore:video**
  - **VideoPremium**
  - **VideoPress**
+ - **videoweed**: VideoWeed
  - **Vidio**
  - **VidLii**
  - **vidme**
@@ -1067,7 +1063,7 @@
  - **VoxMediaVolume**
  - **vpro**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
  - **Vrak**
- - **VRT**: VRT NWS, Flanders News, Flandern Info and Sporza
+ - **VRT**: deredactie.be, sporza.be, cobra.be and cobra.canvas.be
  - **VrtNU**: VrtNU.be
  - **vrv**
  - **vrv:series**
@@ -1097,10 +1093,13 @@
  - **Weibo**
  - **WeiboMobile**
  - **WeiqiTV**: WQTV
+ - **wholecloud**: WholeCloud
  - **Wimp**
  - **Wistia**
  - **wnl**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
  - **WorldStarHipHop**
+ - **wrzuta.pl**
+ - **wrzuta.pl:playlist**
  - **WSJ**: Wall Street Journal
  - **WSJArticle**
  - **WWE**
@@ -1124,13 +1123,10 @@
  - **XVideos**
  - **XXXYMovies**
  - **Yahoo**: Yahoo screen and movies
- - **yahoo:gyao**
- - **yahoo:gyao:player**
  - **YandexDisk**
  - **yandexmusic:album**: Яндекс.Музыка - Альбом
  - **yandexmusic:playlist**: Яндекс.Музыка - Плейлист
  - **yandexmusic:track**: Яндекс.Музыка - Трек
- - **YandexVideo**
  - **YapFiles**
  - **YesJapan**
  - **yinyuetai:video**: 音悦Tai

setup.cfg

@@ -3,4 +3,4 @@ universal = True
 
 [flake8]
 exclude = youtube_dl/extractor/__init__.py,devscripts/buildserver.py,devscripts/lazy_load_template.py,devscripts/make_issue_template.py,setup.py,build,.git,venv
-ignore = E402,E501,E731,E741,W503
+ignore = E402,E501,E731,E741

test/test_InfoExtractor.py

@@ -107,184 +107,6 @@ class TestInfoExtractor(unittest.TestCase):
         self.assertRaises(ExtractorError, self.ie._download_json, uri, None)
         self.assertEqual(self.ie._download_json(uri, None, fatal=False), None)
 
-    def test_parse_html5_media_entries(self):
-        # from https://www.r18.com/
-        # with kpbs in label
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://www.r18.com/',
-                r'''
-                <video id="samplevideo_amateur" class="js-samplevideo video-js vjs-default-skin vjs-big-play-centered" controls preload="auto" width="400" height="225" poster="//pics.r18.com/digital/amateur/mgmr105/mgmr105jp.jpg">
-                    <source id="video_source" src="https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_sm_w.mp4" type="video/mp4"  res="240" label="300kbps">
-                    <source id="video_source" src="https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dm_w.mp4" type="video/mp4"  res="480" label="1000kbps">
-                    <source id="video_source" src="https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dmb_w.mp4" type="video/mp4"  res="740" label="1500kbps">
-                    <p>Your browser does not support the video tag.</p>
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'url': 'https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_sm_w.mp4',
-                    'ext': 'mp4',
-                    'format_id': '300kbps',
-                    'height': 240,
-                    'tbr': 300,
-                }, {
-                    'url': 'https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dm_w.mp4',
-                    'ext': 'mp4',
-                    'format_id': '1000kbps',
-                    'height': 480,
-                    'tbr': 1000,
-                }, {
-                    'url': 'https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dmb_w.mp4',
-                    'ext': 'mp4',
-                    'format_id': '1500kbps',
-                    'height': 740,
-                    'tbr': 1500,
-                }],
-                'thumbnail': '//pics.r18.com/digital/amateur/mgmr105/mgmr105jp.jpg'
-            })
-
-        # from https://www.csfd.cz/
-        # with width and height
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://www.csfd.cz/',
-                r'''
-                <video width="770" height="328" preload="none" controls  poster="https://img.csfd.cz/files/images/film/video/preview/163/344/163344118_748d20.png?h360" >
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327358_eac647.mp4" type="video/mp4" width="640" height="360">
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327360_3d2646.mp4" type="video/mp4" width="1280" height="720">
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327356_91f258.mp4" type="video/mp4" width="1920" height="1080">
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327359_962b4a.webm" type="video/webm" width="640" height="360">
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327361_6feee0.webm" type="video/webm" width="1280" height="720">
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327357_8ab472.webm" type="video/webm" width="1920" height="1080">
-                    <track src="https://video.csfd.cz/files/subtitles/163/344/163344115_4c388b.srt" type="text/x-srt" kind="subtitles" srclang="cs" label="cs">
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327358_eac647.mp4',
-                    'ext': 'mp4',
-                    'width': 640,
-                    'height': 360,
-                }, {
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327360_3d2646.mp4',
-                    'ext': 'mp4',
-                    'width': 1280,
-                    'height': 720,
-                }, {
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327356_91f258.mp4',
-                    'ext': 'mp4',
-                    'width': 1920,
-                    'height': 1080,
-                }, {
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327359_962b4a.webm',
-                    'ext': 'webm',
-                    'width': 640,
-                    'height': 360,
-                }, {
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327361_6feee0.webm',
-                    'ext': 'webm',
-                    'width': 1280,
-                    'height': 720,
-                }, {
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327357_8ab472.webm',
-                    'ext': 'webm',
-                    'width': 1920,
-                    'height': 1080,
-                }],
-                'subtitles': {
-                    'cs': [{'url': 'https://video.csfd.cz/files/subtitles/163/344/163344115_4c388b.srt'}]
-                },
-                'thumbnail': 'https://img.csfd.cz/files/images/film/video/preview/163/344/163344118_748d20.png?h360'
-            })
-
-        # from https://tamasha.com/v/Kkdjw
-        # with height in label
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://tamasha.com/v/Kkdjw',
-                r'''
-                <video crossorigin="anonymous">
-                        <source src="https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4" type="video/mp4" label="AUTO" res="0"/>
-                                <source src="https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4" type="video/mp4"
-                                        label="240p" res="240"/>
-                                <source src="https://s-v2.tamasha.com/statics/videos_file/20/00/Kkdjw_200041c66f657fc967db464d156eafbc1ed9fe6f_n_144.mp4" type="video/mp4"
-                                        label="144p" res="144"/>
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'url': 'https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4',
-                }, {
-                    'url': 'https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4',
-                    'ext': 'mp4',
-                    'format_id': '240p',
-                    'height': 240,
-                }, {
-                    'url': 'https://s-v2.tamasha.com/statics/videos_file/20/00/Kkdjw_200041c66f657fc967db464d156eafbc1ed9fe6f_n_144.mp4',
-                    'ext': 'mp4',
-                    'format_id': '144p',
-                    'height': 144,
-                }]
-            })
-
-        # from https://www.directvnow.com
-        # with data-src
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://www.directvnow.com',
-                r'''
-                <video id="vid1" class="header--video-masked active" muted playsinline>
-                    <source data-src="https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4" type="video/mp4" />
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'ext': 'mp4',
-                    'url': 'https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4',
-                }]
-            })
-
-        # from https://www.directvnow.com
-        # with data-src
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://www.directvnow.com',
-                r'''
-                <video id="vid1" class="header--video-masked active" muted playsinline>
-                    <source data-src="https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4" type="video/mp4" />
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'url': 'https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4',
-                    'ext': 'mp4',
-                }]
-            })
-
-        # from https://www.klarna.com/uk/
-        # with data-video-src
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://www.directvnow.com',
-                r'''
-                <video loop autoplay muted class="responsive-video block-kl__video video-on-medium">
-                    <source src="" data-video-desktop data-video-src="https://www.klarna.com/uk/wp-content/uploads/sites/11/2019/01/KL062_Smooth3_0_DogWalking_5s_920x080_.mp4" type="video/mp4" />
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'url': 'https://www.klarna.com/uk/wp-content/uploads/sites/11/2019/01/KL062_Smooth3_0_DogWalking_5s_920x080_.mp4',
-                    'ext': 'mp4',
-                }],
-            })
-
     def test_extract_jwplayer_data_realworld(self):
         # from http://www.suffolk.edu/sjc/
         expect_dict(
@@ -752,8 +574,7 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                 # Also tests duplicate representation ids, see
                 # https://github.com/ytdl-org/youtube-dl/issues/15111
                 'float_duration',
-                'http://unknown/manifest.mpd',  # mpd_url
-                None,  # mpd_base_url
+                'http://unknown/manifest.mpd',
                 [{
                     'manifest_url': 'http://unknown/manifest.mpd',
                     'ext': 'm4a',
@@ -833,8 +654,7 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
             ), (
                 # https://github.com/ytdl-org/youtube-dl/pull/14844
                 'urls_only',
-                'http://unknown/manifest.mpd',  # mpd_url
-                None,  # mpd_base_url
+                'http://unknown/manifest.mpd',
                 [{
                     'manifest_url': 'http://unknown/manifest.mpd',
                     'ext': 'mp4',
@@ -913,61 +733,15 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                     'width': 1920,
                     'height': 1080,
                 }]
-            ), (
-                # https://github.com/ytdl-org/youtube-dl/issues/20346
-                # Media considered unfragmented even though it contains
-                # Initialization tag
-                'unfragmented',
-                'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',  # mpd_url
-                'https://v.redd.it/hw1x7rcg7zl21',  # mpd_base_url
-                [{
-                    'url': 'https://v.redd.it/hw1x7rcg7zl21/audio',
-                    'manifest_url': 'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',
-                    'ext': 'm4a',
-                    'format_id': 'AUDIO-1',
-                    'format_note': 'DASH audio',
-                    'container': 'm4a_dash',
-                    'acodec': 'mp4a.40.2',
-                    'vcodec': 'none',
-                    'tbr': 129.87,
-                    'asr': 48000,
-
-                }, {
-                    'url': 'https://v.redd.it/hw1x7rcg7zl21/DASH_240',
-                    'manifest_url': 'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',
-                    'ext': 'mp4',
-                    'format_id': 'VIDEO-2',
-                    'format_note': 'DASH video',
-                    'container': 'mp4_dash',
-                    'acodec': 'none',
-                    'vcodec': 'avc1.4d401e',
-                    'tbr': 608.0,
-                    'width': 240,
-                    'height': 240,
-                    'fps': 30,
-                }, {
-                    'url': 'https://v.redd.it/hw1x7rcg7zl21/DASH_360',
-                    'manifest_url': 'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',
-                    'ext': 'mp4',
-                    'format_id': 'VIDEO-1',
-                    'format_note': 'DASH video',
-                    'container': 'mp4_dash',
-                    'acodec': 'none',
-                    'vcodec': 'avc1.4d401e',
-                    'tbr': 804.261,
-                    'width': 360,
-                    'height': 360,
-                    'fps': 30,
-                }]
             )
         ]
 
-        for mpd_file, mpd_url, mpd_base_url, expected_formats in _TEST_CASES:
+        for mpd_file, mpd_url, expected_formats in _TEST_CASES:
             with io.open('./test/testdata/mpd/%s.mpd' % mpd_file,
                          mode='r', encoding='utf-8') as f:
                 formats = self.ie._parse_mpd_formats(
                     compat_etree_fromstring(f.read().encode('utf-8')),
-                    mpd_base_url=mpd_base_url, mpd_url=mpd_url)
+                    mpd_url=mpd_url)
                 self.ie._sort_formats(formats)
                 expect_value(self, formats, expected_formats, None)
 

test/test_aes.py

@@ -44,16 +44,16 @@ class TestAES(unittest.TestCase):
     def test_decrypt_text(self):
         password = intlist_to_bytes(self.key).decode('utf-8')
         encrypted = base64.b64encode(
-            intlist_to_bytes(self.iv[:8])
-            + b'\x17\x15\x93\xab\x8d\x80V\xcdV\xe0\t\xcdo\xc2\xa5\xd8ksM\r\xe27N\xae'
+            intlist_to_bytes(self.iv[:8]) +
+            b'\x17\x15\x93\xab\x8d\x80V\xcdV\xe0\t\xcdo\xc2\xa5\xd8ksM\r\xe27N\xae'
         ).decode('utf-8')
         decrypted = (aes_decrypt_text(encrypted, password, 16))
         self.assertEqual(decrypted, self.secret_msg)
 
         password = intlist_to_bytes(self.key).decode('utf-8')
         encrypted = base64.b64encode(
-            intlist_to_bytes(self.iv[:8])
-            + b'\x0b\xe6\xa4\xd9z\x0e\xb8\xb9\xd0\xd4i_\x85\x1d\x99\x98_\xe5\x80\xe7.\xbf\xa5\x83'
+            intlist_to_bytes(self.iv[:8]) +
+            b'\x0b\xe6\xa4\xd9z\x0e\xb8\xb9\xd0\xd4i_\x85\x1d\x99\x98_\xe5\x80\xe7.\xbf\xa5\x83'
         ).decode('utf-8')
         decrypted = (aes_decrypt_text(encrypted, password, 32))
         self.assertEqual(decrypted, self.secret_msg)

test/test_swfinterp.py

@@ -34,8 +34,8 @@ def _make_testfunc(testfile):
     def test_func(self):
         as_file = os.path.join(TEST_DIR, testfile)
         swf_file = os.path.join(TEST_DIR, test_id + '.swf')
-        if ((not os.path.exists(swf_file))
-                or os.path.getmtime(swf_file) < os.path.getmtime(as_file)):
+        if ((not os.path.exists(swf_file)) or
+                os.path.getmtime(swf_file) < os.path.getmtime(as_file)):
             # Recompile
             try:
                 subprocess.check_call([

test/test_utils.py

@@ -33,13 +33,11 @@ from youtube_dl.utils import (
     ExtractorError,
     find_xpath_attr,
     fix_xml_ampersands,
-    float_or_none,
     get_element_by_class,
     get_element_by_attribute,
     get_elements_by_class,
     get_elements_by_attribute,
     InAdvancePagedList,
-    int_or_none,
     intlist_to_bytes,
     is_html,
     js_to_json,
@@ -57,7 +55,6 @@ from youtube_dl.utils import (
     parse_count,
     parse_iso8601,
     parse_resolution,
-    parse_bitrate,
     pkcs1pad,
     read_batch_urls,
     sanitize_filename,
@@ -73,7 +70,6 @@ from youtube_dl.utils import (
     smuggle_url,
     str_to_int,
     strip_jsonp,
-    strip_or_none,
     timeconvert,
     unescapeHTML,
     unified_strdate,
@@ -184,7 +180,7 @@ class TestUtil(unittest.TestCase):
 
         self.assertEqual(sanitize_filename(
             'ÂÃÄÀÁÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖŐØŒÙÚÛÜŰÝÞßàáâãäåæçèéêëìíîïðñòóôõöőøœùúûüűýþÿ', restricted=True),
-            'AAAAAAAECEEEEIIIIDNOOOOOOOOEUUUUUYTHssaaaaaaaeceeeeiiiionooooooooeuuuuuythy')
+            'AAAAAAAECEEEEIIIIDNOOOOOOOOEUUUUUYPssaaaaaaaeceeeeiiiionooooooooeuuuuuypy')
 
     def test_sanitize_ids(self):
         self.assertEqual(sanitize_filename('_n_cd26wFpw', is_id=True), '_n_cd26wFpw')
@@ -471,21 +467,6 @@ class TestUtil(unittest.TestCase):
             shell_quote(args),
             """ffmpeg -i 'ñ€ß'"'"'.mp4'""" if compat_os_name != 'nt' else '''ffmpeg -i "ñ€ß'.mp4"''')
 
-    def test_float_or_none(self):
-        self.assertEqual(float_or_none('42.42'), 42.42)
-        self.assertEqual(float_or_none('42'), 42.0)
-        self.assertEqual(float_or_none(''), None)
-        self.assertEqual(float_or_none(None), None)
-        self.assertEqual(float_or_none([]), None)
-        self.assertEqual(float_or_none(set()), None)
-
-    def test_int_or_none(self):
-        self.assertEqual(int_or_none('42'), 42)
-        self.assertEqual(int_or_none(''), None)
-        self.assertEqual(int_or_none(None), None)
-        self.assertEqual(int_or_none([]), None)
-        self.assertEqual(int_or_none(set()), None)
-
     def test_str_to_int(self):
         self.assertEqual(str_to_int('123,456'), 123456)
         self.assertEqual(str_to_int('123.456'), 123456)
@@ -753,18 +734,6 @@ class TestUtil(unittest.TestCase):
         d = json.loads(stripped)
         self.assertEqual(d, {'status': 'success'})
 
-    def test_strip_or_none(self):
-        self.assertEqual(strip_or_none(' abc'), 'abc')
-        self.assertEqual(strip_or_none('abc '), 'abc')
-        self.assertEqual(strip_or_none(' abc '), 'abc')
-        self.assertEqual(strip_or_none('\tabc\t'), 'abc')
-        self.assertEqual(strip_or_none('\n\tabc\n\t'), 'abc')
-        self.assertEqual(strip_or_none('abc'), 'abc')
-        self.assertEqual(strip_or_none(''), '')
-        self.assertEqual(strip_or_none(None), None)
-        self.assertEqual(strip_or_none(42), None)
-        self.assertEqual(strip_or_none([]), None)
-
     def test_uppercase_escape(self):
         self.assertEqual(uppercase_escape('aä'), 'aä')
         self.assertEqual(uppercase_escape('\\U0001d550'), '𝕐')
@@ -822,15 +791,6 @@ class TestUtil(unittest.TestCase):
             'vcodec': 'av01.0.05M.08',
             'acodec': 'none',
         })
-        self.assertEqual(parse_codecs('theora, vorbis'), {
-            'vcodec': 'theora',
-            'acodec': 'vorbis',
-        })
-        self.assertEqual(parse_codecs('unknownvcodec, unknownacodec'), {
-            'vcodec': 'unknownvcodec',
-            'acodec': 'unknownacodec',
-        })
-        self.assertEqual(parse_codecs('unknown'), {})
 
     def test_escape_rfc3986(self):
         reserved = "!*'();:@&=+$,/?#[]"
@@ -1070,13 +1030,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(parse_resolution('4k'), {'height': 2160})
         self.assertEqual(parse_resolution('8K'), {'height': 4320})
 
-    def test_parse_bitrate(self):
-        self.assertEqual(parse_bitrate(None), None)
-        self.assertEqual(parse_bitrate(''), None)
-        self.assertEqual(parse_bitrate('300kbps'), 300)
-        self.assertEqual(parse_bitrate('1500kbps'), 1500)
-        self.assertEqual(parse_bitrate('300 kbps'), 300)
-
     def test_version_tuple(self):
         self.assertEqual(version_tuple('1'), (1,))
         self.assertEqual(version_tuple('10.23.344'), (10, 23, 344))

test/testdata/mpd/unfragmented.mpd

@@ -1,28 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
-<MPD mediaPresentationDuration="PT54.915S" minBufferTime="PT1.500S" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" type="static" xmlns="urn:mpeg:dash:schema:mpd:2011">
-    <Period duration="PT54.915S">
-        <AdaptationSet segmentAlignment="true" subsegmentAlignment="true" subsegmentStartsWithSAP="1">
-            <Representation bandwidth="804261" codecs="avc1.4d401e" frameRate="30" height="360" id="VIDEO-1" mimeType="video/mp4" startWithSAP="1" width="360">
-                <BaseURL>DASH_360</BaseURL>
-                <SegmentBase indexRange="915-1114" indexRangeExact="true">
-                    <Initialization range="0-914"/>
-                </SegmentBase>
-            </Representation>
-            <Representation bandwidth="608000" codecs="avc1.4d401e" frameRate="30" height="240" id="VIDEO-2" mimeType="video/mp4" startWithSAP="1" width="240">
-                <BaseURL>DASH_240</BaseURL>
-                <SegmentBase indexRange="913-1112" indexRangeExact="true">
-                    <Initialization range="0-912"/>
-                </SegmentBase>
-            </Representation>
-        </AdaptationSet>
-        <AdaptationSet>
-            <Representation audioSamplingRate="48000" bandwidth="129870" codecs="mp4a.40.2" id="AUDIO-1" mimeType="audio/mp4" startWithSAP="1">
-                <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
-                <BaseURL>audio</BaseURL>
-                <SegmentBase indexRange="832-1007" indexRangeExact="true">
-                    <Initialization range="0-831"/>
-                </SegmentBase>
-            </Representation>
-        </AdaptationSet>
-    </Period>
-</MPD>

youtube_dl/YoutubeDL.py

@@ -309,8 +309,6 @@ class YoutubeDL(object):
     The following options are used by the post processors:
     prefer_ffmpeg:     If False, use avconv instead of ffmpeg if both are available,
                        otherwise prefer ffmpeg.
-    ffmpeg_location:   Location of the ffmpeg/avconv binary; either the path
-                       to the binary or its containing directory.
     postprocessor_args: A list of additional command-line arguments for the
                         postprocessor.
 
@@ -400,9 +398,9 @@ class YoutubeDL(object):
                 else:
                     raise
 
-        if (sys.platform != 'win32'
-                and sys.getfilesystemencoding() in ['ascii', 'ANSI_X3.4-1968']
-                and not params.get('restrictfilenames', False)):
+        if (sys.platform != 'win32' and
+                sys.getfilesystemencoding() in ['ascii', 'ANSI_X3.4-1968'] and
+                not params.get('restrictfilenames', False)):
             # Unicode filesystem API will throw errors (#1474, #13027)
             self.report_warning(
                 'Assuming --restrict-filenames since file system encoding '
@@ -440,9 +438,9 @@ class YoutubeDL(object):
             if re.match(r'^-[0-9A-Za-z_-]{10}$', a)]
         if idxs:
             correct_argv = (
-                ['youtube-dl']
-                + [a for i, a in enumerate(argv) if i not in idxs]
-                + ['--'] + [argv[i] for i in idxs]
+                ['youtube-dl'] +
+                [a for i, a in enumerate(argv) if i not in idxs] +
+                ['--'] + [argv[i] for i in idxs]
             )
             self.report_warning(
                 'Long argument string detected. '
@@ -850,8 +848,8 @@ class YoutubeDL(object):
         if result_type in ('url', 'url_transparent'):
             ie_result['url'] = sanitize_url(ie_result['url'])
             extract_flat = self.params.get('extract_flat', False)
-            if ((extract_flat == 'in_playlist' and 'playlist' in extra_info)
-                    or extract_flat is True):
+            if ((extract_flat == 'in_playlist' and 'playlist' in extra_info) or
+                    extract_flat is True):
                 if self.params.get('forcejson', False):
                     self.to_stdout(json.dumps(ie_result))
                 return ie_result
@@ -1619,9 +1617,9 @@ class YoutubeDL(object):
         # https://github.com/ytdl-org/youtube-dl/issues/10083).
         incomplete_formats = (
             # All formats are video-only or
-            all(f.get('vcodec') != 'none' and f.get('acodec') == 'none' for f in formats)
+            all(f.get('vcodec') != 'none' and f.get('acodec') == 'none' for f in formats) or
             # all formats are audio-only
-            or all(f.get('vcodec') == 'none' and f.get('acodec') != 'none' for f in formats))
+            all(f.get('vcodec') == 'none' and f.get('acodec') != 'none' for f in formats))
 
         ctx = {
             'formats': formats,
@@ -1947,8 +1945,8 @@ class YoutubeDL(object):
                     else:
                         assert fixup_policy in ('ignore', 'never')
 
-                if (info_dict.get('requested_formats') is None
-                        and info_dict.get('container') == 'm4a_dash'):
+                if (info_dict.get('requested_formats') is None and
+                        info_dict.get('container') == 'm4a_dash'):
                     if fixup_policy == 'warn':
                         self.report_warning(
                             '%s: writing DASH m4a. '
@@ -1967,9 +1965,9 @@ class YoutubeDL(object):
                     else:
                         assert fixup_policy in ('ignore', 'never')
 
-                if (info_dict.get('protocol') == 'm3u8_native'
-                        or info_dict.get('protocol') == 'm3u8'
-                        and self.params.get('hls_prefer_native')):
+                if (info_dict.get('protocol') == 'm3u8_native' or
+                        info_dict.get('protocol') == 'm3u8' and
+                        self.params.get('hls_prefer_native')):
                     if fixup_policy == 'warn':
                         self.report_warning('%s: malformed AAC bitstream detected.' % (
                             info_dict['id']))
@@ -1995,10 +1993,10 @@ class YoutubeDL(object):
     def download(self, url_list):
         """Download a given list of URLs."""
         outtmpl = self.params.get('outtmpl', DEFAULT_OUTTMPL)
-        if (len(url_list) > 1
-                and outtmpl != '-'
-                and '%' not in outtmpl
-                and self.params.get('max_downloads') != 1):
+        if (len(url_list) > 1 and
+                outtmpl != '-' and
+                '%' not in outtmpl and
+                self.params.get('max_downloads') != 1):
             raise SameFileError(outtmpl)
 
         for url in url_list:
@@ -2143,8 +2141,8 @@ class YoutubeDL(object):
             if res:
                 res += ', '
             res += '%s container' % fdict['container']
-        if (fdict.get('vcodec') is not None
-                and fdict.get('vcodec') != 'none'):
+        if (fdict.get('vcodec') is not None and
+                fdict.get('vcodec') != 'none'):
             if res:
                 res += ', '
             res += fdict['vcodec']

youtube_dl/__init__.py

@@ -166,8 +166,6 @@ def _real_main(argv=None):
     if opts.max_sleep_interval is not None:
         if opts.max_sleep_interval < 0:
             parser.error('max sleep interval must be positive or 0')
-        if opts.sleep_interval is None:
-            parser.error('min sleep interval must be specified, use --min-sleep-interval')
         if opts.max_sleep_interval < opts.sleep_interval:
             parser.error('max sleep interval must be greater than or equal to min sleep interval')
     else:
@@ -230,14 +228,14 @@ def _real_main(argv=None):
     if opts.allsubtitles and not opts.writeautomaticsub:
         opts.writesubtitles = True
 
-    outtmpl = ((opts.outtmpl is not None and opts.outtmpl)
-               or (opts.format == '-1' and opts.usetitle and '%(title)s-%(id)s-%(format)s.%(ext)s')
-               or (opts.format == '-1' and '%(id)s-%(format)s.%(ext)s')
-               or (opts.usetitle and opts.autonumber and '%(autonumber)s-%(title)s-%(id)s.%(ext)s')
-               or (opts.usetitle and '%(title)s-%(id)s.%(ext)s')
-               or (opts.useid and '%(id)s.%(ext)s')
-               or (opts.autonumber and '%(autonumber)s-%(id)s.%(ext)s')
-               or DEFAULT_OUTTMPL)
+    outtmpl = ((opts.outtmpl is not None and opts.outtmpl) or
+               (opts.format == '-1' and opts.usetitle and '%(title)s-%(id)s-%(format)s.%(ext)s') or
+               (opts.format == '-1' and '%(id)s-%(format)s.%(ext)s') or
+               (opts.usetitle and opts.autonumber and '%(autonumber)s-%(title)s-%(id)s.%(ext)s') or
+               (opts.usetitle and '%(title)s-%(id)s.%(ext)s') or
+               (opts.useid and '%(id)s.%(ext)s') or
+               (opts.autonumber and '%(autonumber)s-%(id)s.%(ext)s') or
+               DEFAULT_OUTTMPL)
     if not os.path.splitext(outtmpl)[1] and opts.extractaudio:
         parser.error('Cannot download a video and extract audio into the same'
                      ' file! Use "{0}.%(ext)s" instead of "{0}" as the output'

youtube_dl/compat.py

@@ -2649,9 +2649,9 @@ else:
 
 try:
     args = shlex.split('中文')
-    assert (isinstance(args, list)
-            and isinstance(args[0], compat_str)
-            and args[0] == '中文')
+    assert (isinstance(args, list) and
+            isinstance(args[0], compat_str) and
+            args[0] == '中文')
     compat_shlex_split = shlex.split
 except (AssertionError, UnicodeEncodeError):
     # Working around shlex issue with unicode strings on some python 2

youtube_dl/downloader/common.py

@@ -176,9 +176,7 @@ class FileDownloader(object):
             return
         speed = float(byte_counter) / elapsed
         if speed > rate_limit:
-            sleep_time = float(byte_counter) / rate_limit - elapsed
-            if sleep_time > 0:
-                time.sleep(sleep_time)
+            time.sleep(max((byte_counter // rate_limit) - elapsed, 0))
 
     def temp_name(self, filename):
         """Returns a temporary filename for the given filename."""
@@ -332,15 +330,15 @@ class FileDownloader(object):
         """
 
         nooverwrites_and_exists = (
-            self.params.get('nooverwrites', False)
-            and os.path.exists(encodeFilename(filename))
+            self.params.get('nooverwrites', False) and
+            os.path.exists(encodeFilename(filename))
         )
 
         if not hasattr(filename, 'write'):
             continuedl_and_exists = (
-                self.params.get('continuedl', True)
-                and os.path.isfile(encodeFilename(filename))
-                and not self.params.get('nopart', False)
+                self.params.get('continuedl', True) and
+                os.path.isfile(encodeFilename(filename)) and
+                not self.params.get('nopart', False)
             )
 
             # Check file already present

youtube_dl/downloader/external.py

@@ -289,7 +289,6 @@ class FFmpegFD(ExternalFD):
             tc_url = info_dict.get('tc_url')
             flash_version = info_dict.get('flash_version')
             live = info_dict.get('rtmp_live', False)
-            conn = info_dict.get('rtmp_conn')
             if player_url is not None:
                 args += ['-rtmp_swfverify', player_url]
             if page_url is not None:
@@ -304,11 +303,6 @@ class FFmpegFD(ExternalFD):
                 args += ['-rtmp_flashver', flash_version]
             if live:
                 args += ['-rtmp_live', 'live']
-            if isinstance(conn, list):
-                for entry in conn:
-                    args += ['-rtmp_conn', entry]
-            elif isinstance(conn, compat_str):
-                args += ['-rtmp_conn', conn]
 
         args += ['-i', url, '-c', 'copy']
 

youtube_dl/downloader/f4m.py

@@ -238,8 +238,8 @@ def write_metadata_tag(stream, metadata):
 
 
 def remove_encrypted_media(media):
-    return list(filter(lambda e: 'drmAdditionalHeaderId' not in e.attrib
-                                 and 'drmAdditionalHeaderSetId' not in e.attrib,
+    return list(filter(lambda e: 'drmAdditionalHeaderId' not in e.attrib and
+                                 'drmAdditionalHeaderSetId' not in e.attrib,
                        media))
 
 
@@ -267,8 +267,8 @@ class F4mFD(FragmentFD):
         media = doc.findall(_add_ns('media'))
         if not media:
             self.report_error('No media found')
-        for e in (doc.findall(_add_ns('drmAdditionalHeader'))
-                  + doc.findall(_add_ns('drmAdditionalHeaderSet'))):
+        for e in (doc.findall(_add_ns('drmAdditionalHeader')) +
+                  doc.findall(_add_ns('drmAdditionalHeaderSet'))):
             # If id attribute is missing it's valid for all media nodes
             # without drmAdditionalHeaderId or drmAdditionalHeaderSetId attribute
             if 'id' not in e.attrib:

youtube_dl/downloader/fragment.py

@@ -219,8 +219,8 @@ class FragmentFD(FileDownloader):
             frag_total_bytes = s.get('total_bytes') or 0
             if not ctx['live']:
                 estimated_size = (
-                    (ctx['complete_frags_downloaded_bytes'] + frag_total_bytes)
-                    / (state['fragment_index'] + 1) * total_frags)
+                    (ctx['complete_frags_downloaded_bytes'] + frag_total_bytes) /
+                    (state['fragment_index'] + 1) * total_frags)
                 state['total_bytes_estimate'] = estimated_size
 
             if s['status'] == 'finished':

youtube_dl/downloader/hls.py

@@ -76,12 +76,12 @@ class HlsFD(FragmentFD):
             return fd.real_download(filename, info_dict)
 
         def is_ad_fragment_start(s):
-            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=ad' in s
-                    or s.startswith('#UPLYNK-SEGMENT') and s.endswith(',ad'))
+            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=ad' in s or
+                    s.startswith('#UPLYNK-SEGMENT') and s.endswith(',ad'))
 
         def is_ad_fragment_end(s):
-            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=master' in s
-                    or s.startswith('#UPLYNK-SEGMENT') and s.endswith(',segment'))
+            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=master' in s or
+                    s.startswith('#UPLYNK-SEGMENT') and s.endswith(',segment'))
 
         media_frags = 0
         ad_frags = 0

youtube_dl/downloader/http.py

@@ -46,8 +46,8 @@ class HttpFD(FileDownloader):
 
         is_test = self.params.get('test', False)
         chunk_size = self._TEST_FILE_SIZE if is_test else (
-            info_dict.get('downloader_options', {}).get('http_chunk_size')
-            or self.params.get('http_chunk_size') or 0)
+            info_dict.get('downloader_options', {}).get('http_chunk_size') or
+            self.params.get('http_chunk_size') or 0)
 
         ctx.open_mode = 'wb'
         ctx.resume_len = 0
@@ -123,11 +123,11 @@ class HttpFD(FileDownloader):
                                 content_len = int_or_none(content_range_m.group(3))
                                 accept_content_len = (
                                     # Non-chunked download
-                                    not ctx.chunk_size
+                                    not ctx.chunk_size or
                                     # Chunked download and requested piece or
                                     # its part is promised to be served
-                                    or content_range_end == range_end
-                                    or content_len < range_end)
+                                    content_range_end == range_end or
+                                    content_len < range_end)
                                 if accept_content_len:
                                     ctx.data_len = content_len
                                     return
@@ -152,8 +152,8 @@ class HttpFD(FileDownloader):
                             raise
                     else:
                         # Examine the reported length
-                        if (content_length is not None
-                                and (ctx.resume_len - 100 < int(content_length) < ctx.resume_len + 100)):
+                        if (content_length is not None and
+                                (ctx.resume_len - 100 < int(content_length) < ctx.resume_len + 100)):
                             # The file had already been fully downloaded.
                             # Explanation to the above condition: in issue #175 it was revealed that
                             # YouTube sometimes adds or removes a few bytes from the end of the file,

youtube_dl/extractor/addanime.py

@@ -59,9 +59,9 @@ class AddAnimeIE(InfoExtractor):
             parsed_url = compat_urllib_parse_urlparse(url)
             av_val = av_res + len(parsed_url.netloc)
             confirm_url = (
-                parsed_url.scheme + '://' + parsed_url.netloc
-                + action + '?'
-                + compat_urllib_parse_urlencode({
+                parsed_url.scheme + '://' + parsed_url.netloc +
+                action + '?' +
+                compat_urllib_parse_urlencode({
                     'jschl_vc': vc, 'jschl_answer': compat_str(av_val)}))
             self._download_webpage(
                 confirm_url, video_id,

youtube_dl/extractor/adn.py

@@ -21,6 +21,7 @@ from ..utils import (
     intlist_to_bytes,
     long_to_bytes,
     pkcs1pad,
+    srt_subtitles_timecode,
     strip_or_none,
     urljoin,
 )
@@ -41,18 +42,6 @@ class ADNIE(InfoExtractor):
     }
     _BASE_URL = 'http://animedigitalnetwork.fr'
     _RSA_KEY = (0xc35ae1e4356b65a73b551493da94b8cb443491c0aa092a357a5aee57ffc14dda85326f42d716e539a34542a0d3f363adf16c5ec222d713d5997194030ee2e4f0d1fb328c01a81cf6868c090d50de8e169c6b13d1675b9eeed1cbc51e1fffca9b38af07f37abd790924cd3bee59d0257cfda4fe5f3f0534877e21ce5821447d1b, 65537)
-    _POS_ALIGN_MAP = {
-        'start': 1,
-        'end': 3,
-    }
-    _LINE_ALIGN_MAP = {
-        'middle': 8,
-        'end': 4,
-    }
-
-    @staticmethod
-    def _ass_subtitles_timecode(seconds):
-        return '%01d:%02d:%02d.%02d' % (seconds / 3600, (seconds % 3600) / 60, seconds % 60, (seconds % 1) * 100)
 
     def _get_subtitles(self, sub_path, video_id):
         if not sub_path:
@@ -60,20 +49,14 @@ class ADNIE(InfoExtractor):
 
         enc_subtitles = self._download_webpage(
             urljoin(self._BASE_URL, sub_path),
-            video_id, 'Downloading subtitles location', fatal=False) or '{}'
-        subtitle_location = (self._parse_json(enc_subtitles, video_id, fatal=False) or {}).get('location')
-        if subtitle_location:
-            enc_subtitles = self._download_webpage(
-                urljoin(self._BASE_URL, subtitle_location),
-                video_id, 'Downloading subtitles data', fatal=False,
-                headers={'Origin': 'https://animedigitalnetwork.fr'})
+            video_id, fatal=False)
         if not enc_subtitles:
             return None
 
         # http://animedigitalnetwork.fr/components/com_vodvideo/videojs/adn-vjs.min.js
         dec_subtitles = intlist_to_bytes(aes_cbc_decrypt(
             bytes_to_intlist(compat_b64decode(enc_subtitles[24:])),
-            bytes_to_intlist(binascii.unhexlify(self._K + '4b8ef13ec1872730')),
+            bytes_to_intlist(binascii.unhexlify(self._K + '9032ad7083106400')),
             bytes_to_intlist(compat_b64decode(enc_subtitles[:24]))
         ))
         subtitles_json = self._parse_json(
@@ -84,27 +67,23 @@ class ADNIE(InfoExtractor):
 
         subtitles = {}
         for sub_lang, sub in subtitles_json.items():
-            ssa = '''[Script Info]
-ScriptType:V4.00
-[V4 Styles]
-Format: Name,Fontname,Fontsize,PrimaryColour,SecondaryColour,TertiaryColour,BackColour,Bold,Italic,BorderStyle,Outline,Shadow,Alignment,MarginL,MarginR,MarginV,AlphaLevel,Encoding
-Style: Default,Arial,18,16777215,16777215,16777215,0,-1,0,1,1,0,2,20,20,20,0,0
-[Events]
-Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
-            for current in sub:
-                start, end, text, line_align, position_align = (
+            srt = ''
+            for num, current in enumerate(sub):
+                start, end, text = (
                     float_or_none(current.get('startTime')),
                     float_or_none(current.get('endTime')),
-                    current.get('text'), current.get('lineAlign'),
-                    current.get('positionAlign'))
+                    current.get('text'))
                 if start is None or end is None or text is None:
                     continue
-                alignment = self._POS_ALIGN_MAP.get(position_align, 2) + self._LINE_ALIGN_MAP.get(line_align, 0)
-                ssa += os.linesep + 'Dialogue: Marked=0,%s,%s,Default,,0,0,0,,%s%s' % (
-                    self._ass_subtitles_timecode(start),
-                    self._ass_subtitles_timecode(end),
-                    '{\\a%d}' % alignment if alignment != 2 else '',
-                    text.replace('\n', '\\N').replace('<i>', '{\\i1}').replace('</i>', '{\\i0}'))
+                srt += os.linesep.join(
+                    (
+                        '%d' % num,
+                        '%s --> %s' % (
+                            srt_subtitles_timecode(start),
+                            srt_subtitles_timecode(end)),
+                        text,
+                        os.linesep,
+                    ))
 
             if sub_lang == 'vostf':
                 sub_lang = 'fr'
@@ -112,8 +91,8 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
                 'ext': 'json',
                 'data': json.dumps(sub),
             }, {
-                'ext': 'ssa',
-                'data': ssa,
+                'ext': 'srt',
+                'data': srt,
             }])
         return subtitles
 
@@ -121,15 +100,7 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
         video_id = self._match_id(url)
         webpage = self._download_webpage(url, video_id)
         player_config = self._parse_json(self._search_regex(
-            r'playerConfig\s*=\s*({.+});', webpage,
-            'player config', default='{}'), video_id, fatal=False)
-        if not player_config:
-            config_url = urljoin(self._BASE_URL, self._search_regex(
-                r'(?:id="player"|class="[^"]*adn-player-container[^"]*")[^>]+data-url="([^"]+)"',
-                webpage, 'config url'))
-            player_config = self._download_json(
-                config_url, video_id,
-                'Downloading player config JSON metadata')['player']
+            r'playerConfig\s*=\s*({.+});', webpage, 'player config'), video_id)
 
         video_info = {}
         video_info_str = self._search_regex(
@@ -158,15 +129,12 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
             encrypted_message = long_to_bytes(pow(bytes_to_long(padded_message), e, n))
             authorization = base64.b64encode(encrypted_message).decode()
             links_data = self._download_json(
-                urljoin(self._BASE_URL, links_url), video_id,
-                'Downloading links JSON metadata', headers={
+                urljoin(self._BASE_URL, links_url), video_id, headers={
                     'Authorization': 'Bearer ' + authorization,
                 })
             links = links_data.get('links') or {}
             metas = metas or links_data.get('meta') or {}
-            sub_path = sub_path or links_data.get('subtitles') or \
-                'index.php?option=com_vodapi&task=subtitles.getJSON&format=json&id=' + video_id
-            sub_path += '&token=' + token
+            sub_path = (sub_path or links_data.get('subtitles')) + '&token=' + token
             error = links_data.get('error')
         title = metas.get('title') or video_info['title']
 
@@ -174,11 +142,9 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
         for format_id, qualities in links.items():
             if not isinstance(qualities, dict):
                 continue
-            for quality, load_balancer_url in qualities.items():
+            for load_balancer_url in qualities.values():
                 load_balancer_data = self._download_json(
-                    load_balancer_url, video_id,
-                    'Downloading %s %s JSON metadata' % (format_id, quality),
-                    fatal=False) or {}
+                    load_balancer_url, video_id, fatal=False) or {}
                 m3u8_url = load_balancer_data.get('location')
                 if not m3u8_url:
                     continue

youtube_dl/extractor/adobeconnect.py

@@ -1,37 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..compat import (
-    compat_parse_qs,
-    compat_urlparse,
-)
-
-
-class AdobeConnectIE(InfoExtractor):
-    _VALID_URL = r'https?://\w+\.adobeconnect\.com/(?P<id>[\w-]+)'
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        title = self._html_search_regex(r'<title>(.+?)</title>', webpage, 'title')
-        qs = compat_parse_qs(self._search_regex(r"swfUrl\s*=\s*'([^']+)'", webpage, 'swf url').split('?')[1])
-        is_live = qs.get('isLive', ['false'])[0] == 'true'
-        formats = []
-        for con_string in qs['conStrings'][0].split(','):
-            formats.append({
-                'format_id': con_string.split('://')[0],
-                'app': compat_urlparse.quote('?' + con_string.split('?')[1] + 'flvplayerapp/' + qs['appInstance'][0]),
-                'ext': 'flv',
-                'play_path': 'mp4:' + qs['streamName'][0],
-                'rtmp_conn': 'S:' + qs['ticket'][0],
-                'rtmp_live': is_live,
-                'url': con_string,
-            })
-
-        return {
-            'id': video_id,
-            'title': self._live_title(title) if is_live else title,
-            'formats': formats,
-            'is_live': is_live,
-        }

youtube_dl/extractor/adultswim.py

@@ -1,19 +1,13 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import json
 import re
 
 from .turner import TurnerBaseIE
 from ..utils import (
-    determine_ext,
-    float_or_none,
     int_or_none,
-    mimetype2ext,
-    parse_age_limit,
-    parse_iso8601,
     strip_or_none,
-    try_get,
+    url_or_none,
 )
 
 
@@ -27,8 +21,8 @@ class AdultSwimIE(TurnerBaseIE):
             'ext': 'mp4',
             'title': 'Rick and Morty - Pilot',
             'description': 'Rick moves in with his daughter\'s family and establishes himself as a bad influence on his grandson, Morty.',
-            'timestamp': 1543294800,
-            'upload_date': '20181127',
+            'timestamp': 1493267400,
+            'upload_date': '20170427',
         },
         'params': {
             # m3u8 download
@@ -49,7 +43,6 @@ class AdultSwimIE(TurnerBaseIE):
             # m3u8 download
             'skip_download': True,
         },
-        'skip': '404 Not Found',
     }, {
         'url': 'http://www.adultswim.com/videos/decker/inside-decker-a-new-hero/',
         'info_dict': {
@@ -68,9 +61,9 @@ class AdultSwimIE(TurnerBaseIE):
     }, {
         'url': 'http://www.adultswim.com/videos/attack-on-titan',
         'info_dict': {
-            'id': 'attack-on-titan',
+            'id': 'b7A69dzfRzuaXIECdxW8XQ',
             'title': 'Attack on Titan',
-            'description': 'md5:41caa9416906d90711e31dc00cb7db7e',
+            'description': 'md5:6c8e003ea0777b47013e894767f5e114',
         },
         'playlist_mincount': 12,
     }, {
@@ -85,118 +78,83 @@ class AdultSwimIE(TurnerBaseIE):
             # m3u8 download
             'skip_download': True,
         },
-        'skip': '404 Not Found',
     }]
 
     def _real_extract(self, url):
         show_path, episode_path = re.match(self._VALID_URL, url).groups()
         display_id = episode_path or show_path
-        query = '''query {
-  getShowBySlug(slug:"%s") {
-    %%s
-  }
-}''' % show_path
-        if episode_path:
-            query = query % '''title
-    getVideoBySlug(slug:"%s") {
-      _id
-      auth
-      description
-      duration
-      episodeNumber
-      launchDate
-      mediaID
-      seasonNumber
-      poster
-      title
-      tvRating
-    }''' % episode_path
-            ['getVideoBySlug']
-        else:
-            query = query % '''metaDescription
-    title
-    videos(first:1000,sort:["episode_number"]) {
-      edges {
-        node {
-           _id
-           slug
-        }
-      }
-    }'''
-        show_data = self._download_json(
-            'https://www.adultswim.com/api/search', display_id,
-            data=json.dumps({'query': query}).encode(),
-            headers={'Content-Type': 'application/json'})['data']['getShowBySlug']
-        if episode_path:
-            video_data = show_data['getVideoBySlug']
-            video_id = video_data['_id']
-            episode_title = title = video_data['title']
-            series = show_data.get('title')
-            if series:
-                title = '%s - %s' % (series, title)
-            info = {
-                'id': video_id,
-                'title': title,
-                'description': strip_or_none(video_data.get('description')),
-                'duration': float_or_none(video_data.get('duration')),
-                'formats': [],
-                'subtitles': {},
-                'age_limit': parse_age_limit(video_data.get('tvRating')),
-                'thumbnail': video_data.get('poster'),
-                'timestamp': parse_iso8601(video_data.get('launchDate')),
-                'series': series,
-                'season_number': int_or_none(video_data.get('seasonNumber')),
-                'episode': episode_title,
-                'episode_number': int_or_none(video_data.get('episodeNumber')),
-            }
+        webpage = self._download_webpage(url, display_id)
+        initial_data = self._parse_json(self._search_regex(
+            r'AS_INITIAL_DATA(?:__)?\s*=\s*({.+?});',
+            webpage, 'initial data'), display_id)
 
-            auth = video_data.get('auth')
-            media_id = video_data.get('mediaID')
-            if media_id:
-                info.update(self._extract_ngtv_info(media_id, {
-                    # CDN_TOKEN_APP_ID from:
-                    # https://d2gg02c3xr550i.cloudfront.net/assets/asvp.e9c8bef24322d060ef87.bundle.js
-                    'appId': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhcHBJZCI6ImFzLXR2ZS1kZXNrdG9wLXB0enQ2bSIsInByb2R1Y3QiOiJ0dmUiLCJuZXR3b3JrIjoiYXMiLCJwbGF0Zm9ybSI6ImRlc2t0b3AiLCJpYXQiOjE1MzI3MDIyNzl9.BzSCk-WYOZ2GMCIaeVb8zWnzhlgnXuJTCu0jGp_VaZE',
-                }, {
-                    'url': url,
-                    'site_name': 'AdultSwim',
-                    'auth_required': auth,
-                }))
+        is_stream = show_path == 'streams'
+        if is_stream:
+            if not episode_path:
+                episode_path = 'live-stream'
 
-            if not auth:
-                extract_data = self._download_json(
-                    'https://www.adultswim.com/api/shows/v1/videos/' + video_id,
-                    video_id, query={'fields': 'stream'}, fatal=False) or {}
-                assets = try_get(extract_data, lambda x: x['data']['video']['stream']['assets'], list) or []
-                for asset in assets:
-                    asset_url = asset.get('url')
-                    if not asset_url:
+            video_data = next(stream for stream_path, stream in initial_data['streams'].items() if stream_path == episode_path)
+            video_id = video_data.get('stream')
+
+            if not video_id:
+                entries = []
+                for episode in video_data.get('archiveEpisodes', []):
+                    episode_url = url_or_none(episode.get('url'))
+                    if not episode_url:
                         continue
-                    ext = determine_ext(asset_url, mimetype2ext(asset.get('mime_type')))
-                    if ext == 'm3u8':
-                        info['formats'].extend(self._extract_m3u8_formats(
-                            asset_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
-                    elif ext == 'f4m':
-                        continue
-                        # info['formats'].extend(self._extract_f4m_formats(
-                        #     asset_url, video_id, f4m_id='hds', fatal=False))
-                    elif ext in ('scc', 'ttml', 'vtt'):
-                        info['subtitles'].setdefault('en', []).append({
-                            'url': asset_url,
-                        })
-            self._sort_formats(info['formats'])
-
-            return info
+                    entries.append(self.url_result(
+                        episode_url, 'AdultSwim', episode.get('id')))
+                return self.playlist_result(
+                    entries, video_data.get('id'), video_data.get('title'),
+                    strip_or_none(video_data.get('description')))
         else:
-            entries = []
-            for edge in show_data.get('videos', {}).get('edges', []):
-                video = edge.get('node') or {}
-                slug = video.get('slug')
-                if not slug:
-                    continue
-                entries.append(self.url_result(
-                    'http://adultswim.com/videos/%s/%s' % (show_path, slug),
-                    'AdultSwim', video.get('_id')))
-            return self.playlist_result(
-                entries, show_path, show_data.get('title'),
-                strip_or_none(show_data.get('metaDescription')))
+            show_data = initial_data['show']
+
+            if not episode_path:
+                entries = []
+                for video in show_data.get('videos', []):
+                    slug = video.get('slug')
+                    if not slug:
+                        continue
+                    entries.append(self.url_result(
+                        'http://adultswim.com/videos/%s/%s' % (show_path, slug),
+                        'AdultSwim', video.get('id')))
+                return self.playlist_result(
+                    entries, show_data.get('id'), show_data.get('title'),
+                    strip_or_none(show_data.get('metadata', {}).get('description')))
+
+            video_data = show_data['sluggedVideo']
+            video_id = video_data['id']
+
+        info = self._extract_cvp_info(
+            'http://www.adultswim.com/videos/api/v0/assets?platform=desktop&id=' + video_id,
+            video_id, {
+                'secure': {
+                    'media_src': 'http://androidhls-secure.cdn.turner.com/adultswim/big',
+                    'tokenizer_src': 'http://www.adultswim.com/astv/mvpd/processors/services/token_ipadAdobe.do',
+                },
+            }, {
+                'url': url,
+                'site_name': 'AdultSwim',
+                'auth_required': video_data.get('auth'),
+            })
+
+        info.update({
+            'id': video_id,
+            'display_id': display_id,
+            'description': info.get('description') or strip_or_none(video_data.get('description')),
+        })
+        if not is_stream:
+            info.update({
+                'duration': info.get('duration') or int_or_none(video_data.get('duration')),
+                'timestamp': info.get('timestamp') or int_or_none(video_data.get('launch_date')),
+                'season_number': info.get('season_number') or int_or_none(video_data.get('season_number')),
+                'episode': info['title'],
+                'episode_number': info.get('episode_number') or int_or_none(video_data.get('episode_number')),
+            })
+
+            info['series'] = video_data.get('collection_title') or info.get('series')
+            if info['series'] and info['series'] != info['title']:
+                info['title'] = '%s - %s' % (info['series'], info['title'])
+
+        return info

youtube_dl/extractor/aenetworks.py

@@ -1,15 +1,14 @@
-# coding: utf-8
 from __future__ import unicode_literals
 
 import re
 
 from .theplatform import ThePlatformIE
 from ..utils import (
-    extract_attributes,
-    ExtractorError,
-    int_or_none,
     smuggle_url,
     update_url_query,
+    unescapeHTML,
+    extract_attributes,
+    get_element_by_attribute,
 )
 from ..compat import (
     compat_urlparse,
@@ -20,43 +19,6 @@ class AENetworksBaseIE(ThePlatformIE):
     _THEPLATFORM_KEY = 'crazyjava'
     _THEPLATFORM_SECRET = 's3cr3t'
 
-    def _extract_aen_smil(self, smil_url, video_id, auth=None):
-        query = {'mbr': 'true'}
-        if auth:
-            query['auth'] = auth
-        TP_SMIL_QUERY = [{
-            'assetTypes': 'high_video_ak',
-            'switch': 'hls_high_ak'
-        }, {
-            'assetTypes': 'high_video_s3'
-        }, {
-            'assetTypes': 'high_video_s3',
-            'switch': 'hls_ingest_fastly'
-        }]
-        formats = []
-        subtitles = {}
-        last_e = None
-        for q in TP_SMIL_QUERY:
-            q.update(query)
-            m_url = update_url_query(smil_url, q)
-            m_url = self._sign_url(m_url, self._THEPLATFORM_KEY, self._THEPLATFORM_SECRET)
-            try:
-                tp_formats, tp_subtitles = self._extract_theplatform_smil(
-                    m_url, video_id, 'Downloading %s SMIL data' % (q.get('switch') or q['assetTypes']))
-            except ExtractorError as e:
-                last_e = e
-                continue
-            formats.extend(tp_formats)
-            subtitles = self._merge_subtitles(subtitles, tp_subtitles)
-        if last_e and not formats:
-            raise last_e
-        self._sort_formats(formats)
-        return {
-            'id': video_id,
-            'formats': formats,
-            'subtitles': subtitles,
-        }
-
 
 class AENetworksIE(AENetworksBaseIE):
     IE_NAME = 'aenetworks'
@@ -71,25 +33,22 @@ class AENetworksIE(AENetworksBaseIE):
                         (?:
                             shows/(?P<show_path>[^/]+(?:/[^/]+){0,2})|
                             movies/(?P<movie_display_id>[^/]+)(?:/full-movie)?|
-                            specials/(?P<special_display_id>[^/]+)/(?:full-special|preview-)|
+                            specials/(?P<special_display_id>[^/]+)/full-special|
                             collections/[^/]+/(?P<collection_display_id>[^/]+)
                         )
                     '''
     _TESTS = [{
         'url': 'http://www.history.com/shows/mountain-men/season-1/episode-1',
+        'md5': 'a97a65f7e823ae10e9244bc5433d5fe6',
         'info_dict': {
             'id': '22253814',
             'ext': 'mp4',
-            'title': 'Winter is Coming',
+            'title': 'Winter Is Coming',
             'description': 'md5:641f424b7a19d8e24f26dea22cf59d74',
             'timestamp': 1338306241,
             'upload_date': '20120529',
             'uploader': 'AENE-NEW',
         },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
         'add_ie': ['ThePlatform'],
     }, {
         'url': 'http://www.history.com/shows/ancient-aliens/season-1',
@@ -125,9 +84,6 @@ class AENetworksIE(AENetworksBaseIE):
     }, {
         'url': 'https://www.historyvault.com/collections/america-the-story-of-us/westward',
         'only_matching': True
-    }, {
-        'url': 'https://www.aetv.com/specials/hunting-jonbenets-killer-the-untold-story/preview-hunting-jonbenets-killer-the-untold-story',
-        'only_matching': True
     }]
     _DOMAIN_TO_REQUESTOR_ID = {
         'history.com': 'HISTORY',
@@ -168,6 +124,11 @@ class AENetworksIE(AENetworksBaseIE):
                 return self.playlist_result(
                     entries, self._html_search_meta('aetn:SeasonId', webpage))
 
+        query = {
+            'mbr': 'true',
+            'assetTypes': 'high_video_ak',
+            'switch': 'hls_high_ak',
+        }
         video_id = self._html_search_meta('aetn:VideoID', webpage)
         media_url = self._search_regex(
             [r"media_url\s*=\s*'(?P<url>[^']+)'",
@@ -177,39 +138,64 @@ class AENetworksIE(AENetworksBaseIE):
         theplatform_metadata = self._download_theplatform_metadata(self._search_regex(
             r'https?://link\.theplatform\.com/s/([^?]+)', media_url, 'theplatform_path'), video_id)
         info = self._parse_theplatform_metadata(theplatform_metadata)
-        auth = None
         if theplatform_metadata.get('AETN$isBehindWall'):
             requestor_id = self._DOMAIN_TO_REQUESTOR_ID[domain]
             resource = self._get_mvpd_resource(
                 requestor_id, theplatform_metadata['title'],
                 theplatform_metadata.get('AETN$PPL_pplProgramId') or theplatform_metadata.get('AETN$PPL_pplProgramId_OLD'),
                 theplatform_metadata['ratings'][0]['rating'])
-            auth = self._extract_mvpd_auth(
+            query['auth'] = self._extract_mvpd_auth(
                 url, video_id, requestor_id, resource)
         info.update(self._search_json_ld(webpage, video_id, fatal=False))
-        info.update(self._extract_aen_smil(media_url, video_id, auth))
+        media_url = update_url_query(media_url, query)
+        media_url = self._sign_url(media_url, self._THEPLATFORM_KEY, self._THEPLATFORM_SECRET)
+        formats, subtitles = self._extract_theplatform_smil(media_url, video_id)
+        self._sort_formats(formats)
+        info.update({
+            'id': video_id,
+            'formats': formats,
+            'subtitles': subtitles,
+        })
         return info
 
 
 class HistoryTopicIE(AENetworksBaseIE):
     IE_NAME = 'history:topic'
     IE_DESC = 'History.com Topic'
-    _VALID_URL = r'https?://(?:www\.)?history\.com/topics/[^/]+/(?P<id>[\w+-]+?)-video'
+    _VALID_URL = r'https?://(?:www\.)?history\.com/topics/(?:[^/]+/)?(?P<topic_id>[^/]+)(?:/[^/]+(?:/(?P<video_display_id>[^/?#]+))?)?'
     _TESTS = [{
-        'url': 'https://www.history.com/topics/valentines-day/history-of-valentines-day-video',
+        'url': 'http://www.history.com/topics/valentines-day/history-of-valentines-day/videos/bet-you-didnt-know-valentines-day?m=528e394da93ae&s=undefined&f=1&free=false',
         'info_dict': {
             'id': '40700995724',
             'ext': 'mp4',
-            'title': "History of Valentine’s Day",
+            'title': "Bet You Didn't Know: Valentine's Day",
             'description': 'md5:7b57ea4829b391995b405fa60bd7b5f7',
             'timestamp': 1375819729,
             'upload_date': '20130806',
+            'uploader': 'AENE-NEW',
         },
         'params': {
             # m3u8 download
             'skip_download': True,
         },
         'add_ie': ['ThePlatform'],
+    }, {
+        'url': 'http://www.history.com/topics/world-war-i/world-war-i-history/videos',
+        'info_dict':
+        {
+            'id': 'world-war-i-history',
+            'title': 'World War I History',
+        },
+        'playlist_mincount': 23,
+    }, {
+        'url': 'http://www.history.com/topics/world-war-i-history/videos',
+        'only_matching': True,
+    }, {
+        'url': 'http://www.history.com/topics/world-war-i/world-war-i-history',
+        'only_matching': True,
+    }, {
+        'url': 'http://www.history.com/topics/world-war-i/world-war-i-history/speeches',
+        'only_matching': True,
     }]
 
     def theplatform_url_result(self, theplatform_url, video_id, query):
@@ -229,19 +215,27 @@ class HistoryTopicIE(AENetworksBaseIE):
         }
 
     def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-        video_id = self._search_regex(
-            r'<phoenix-iframe[^>]+src="[^"]+\btpid=(\d+)', webpage, 'tpid')
-        result = self._download_json(
-            'https://feeds.video.aetnd.com/api/v2/history/videos',
-            video_id, query={'filter[id]': video_id})['results'][0]
-        title = result['title']
-        info = self._extract_aen_smil(result['publicUrl'], video_id)
-        info.update({
-            'title': title,
-            'description': result.get('description'),
-            'duration': int_or_none(result.get('duration')),
-            'timestamp': int_or_none(result.get('added'), 1000),
-        })
-        return info
+        topic_id, video_display_id = re.match(self._VALID_URL, url).groups()
+        if video_display_id:
+            webpage = self._download_webpage(url, video_display_id)
+            release_url, video_id = re.search(r"_videoPlayer.play\('([^']+)'\s*,\s*'[^']+'\s*,\s*'(\d+)'\)", webpage).groups()
+            release_url = unescapeHTML(release_url)
+
+            return self.theplatform_url_result(
+                release_url, video_id, {
+                    'mbr': 'true',
+                    'switch': 'hls',
+                    'assetTypes': 'high_video_ak',
+                })
+        else:
+            webpage = self._download_webpage(url, topic_id)
+            entries = []
+            for episode_item in re.findall(r'<a.+?data-release-url="[^"]+"[^>]*>', webpage):
+                video_attributes = extract_attributes(episode_item)
+                entries.append(self.theplatform_url_result(
+                    video_attributes['data-release-url'], video_attributes['data-id'], {
+                        'mbr': 'true',
+                        'switch': 'hls',
+                        'assetTypes': 'high_video_ak',
+                    }))
+            return self.playlist_result(entries, topic_id, get_element_by_attribute('class', 'show-title', webpage))

youtube_dl/extractor/anitube.py

@@ -0,0 +1,30 @@
+from __future__ import unicode_literals
+
+from .nuevo import NuevoBaseIE
+
+
+class AnitubeIE(NuevoBaseIE):
+    IE_NAME = 'anitube.se'
+    _VALID_URL = r'https?://(?:www\.)?anitube\.se/video/(?P<id>\d+)'
+
+    _TEST = {
+        'url': 'http://www.anitube.se/video/36621',
+        'md5': '59d0eeae28ea0bc8c05e7af429998d43',
+        'info_dict': {
+            'id': '36621',
+            'ext': 'mp4',
+            'title': 'Recorder to Randoseru 01',
+            'duration': 180.19,
+        },
+        'skip': 'Blocked in the US',
+    }
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+
+        webpage = self._download_webpage(url, video_id)
+        key = self._search_regex(
+            r'src=["\']https?://[^/]+/embed/([A-Za-z0-9_-]+)', webpage, 'key')
+
+        return self._extract_nuevo(
+            'http://www.anitube.se/nuevo/econfig.php?key=%s' % key, video_id)

youtube_dl/extractor/anysex.py

@@ -0,0 +1,61 @@
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    parse_duration,
+    int_or_none,
+)
+
+
+class AnySexIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?anysex\.com/(?P<id>\d+)'
+    _TEST = {
+        'url': 'http://anysex.com/156592/',
+        'md5': '023e9fbb7f7987f5529a394c34ad3d3d',
+        'info_dict': {
+            'id': '156592',
+            'ext': 'mp4',
+            'title': 'Busty and sexy blondie in her bikini strips for you',
+            'description': 'md5:de9e418178e2931c10b62966474e1383',
+            'categories': ['Erotic'],
+            'duration': 270,
+            'age_limit': 18,
+        }
+    }
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        video_id = mobj.group('id')
+
+        webpage = self._download_webpage(url, video_id)
+
+        video_url = self._html_search_regex(r"video_url\s*:\s*'([^']+)'", webpage, 'video URL')
+
+        title = self._html_search_regex(r'<title>(.*?)</title>', webpage, 'title')
+        description = self._html_search_regex(
+            r'<div class="description"[^>]*>([^<]+)</div>', webpage, 'description', fatal=False)
+        thumbnail = self._html_search_regex(
+            r'preview_url\s*:\s*\'(.*?)\'', webpage, 'thumbnail', fatal=False)
+
+        categories = re.findall(
+            r'<a href="http://anysex\.com/categories/[^"]+" title="[^"]*">([^<]+)</a>', webpage)
+
+        duration = parse_duration(self._search_regex(
+            r'<b>Duration:</b> (?:<q itemprop="duration">)?(\d+:\d+)', webpage, 'duration', fatal=False))
+        view_count = int_or_none(self._html_search_regex(
+            r'<b>Views:</b> (\d+)', webpage, 'view count', fatal=False))
+
+        return {
+            'id': video_id,
+            'url': video_url,
+            'ext': 'mp4',
+            'title': title,
+            'description': description,
+            'thumbnail': thumbnail,
+            'categories': categories,
+            'duration': duration,
+            'view_count': view_count,
+            'age_limit': 18,
+        }

youtube_dl/extractor/aol.py

@@ -4,10 +4,6 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from ..compat import (
-    compat_parse_qs,
-    compat_urllib_parse_urlparse,
-)
 from ..utils import (
     ExtractorError,
     int_or_none,
@@ -16,12 +12,12 @@ from ..utils import (
 
 
 class AolIE(InfoExtractor):
-    IE_NAME = 'aol.com'
-    _VALID_URL = r'(?:aol-video:|https?://(?:www\.)?aol\.(?:com|ca|co\.uk|de|jp)/video/(?:[^/]+/)*)(?P<id>[0-9a-f]+)'
+    IE_NAME = 'on.aol.com'
+    _VALID_URL = r'(?:aol-video:|https?://(?:(?:www|on)\.)?aol\.com/(?:[^/]+/)*(?:[^/?#&]+-)?)(?P<id>[^/?#&]+)'
 
     _TESTS = [{
         # video with 5min ID
-        'url': 'https://www.aol.com/video/view/u-s--official-warns-of-largest-ever-irs-phone-scam/518167793/',
+        'url': 'http://on.aol.com/video/u-s--official-warns-of-largest-ever-irs-phone-scam-518167793?icid=OnHomepageC2Wide_MustSee_Img',
         'md5': '18ef68f48740e86ae94b98da815eec42',
         'info_dict': {
             'id': '518167793',
@@ -38,7 +34,7 @@ class AolIE(InfoExtractor):
         }
     }, {
         # video with vidible ID
-        'url': 'https://www.aol.com/video/view/netflix-is-raising-rates/5707d6b8e4b090497b04f706/',
+        'url': 'http://www.aol.com/video/view/netflix-is-raising-rates/5707d6b8e4b090497b04f706/',
         'info_dict': {
             'id': '5707d6b8e4b090497b04f706',
             'ext': 'mp4',
@@ -53,29 +49,17 @@ class AolIE(InfoExtractor):
             'skip_download': True,
         }
     }, {
-        'url': 'https://www.aol.com/video/view/park-bench-season-2-trailer/559a1b9be4b0c3bfad3357a7/',
+        'url': 'http://on.aol.com/partners/abc-551438d309eab105804dbfe8/sneak-peek-was-haley-really-framed-570eaebee4b0448640a5c944',
         'only_matching': True,
     }, {
-        'url': 'https://www.aol.com/video/view/donald-trump-spokeswoman-tones-down-megyn-kelly-attacks/519442220/',
+        'url': 'http://on.aol.com/shows/park-bench-shw518173474-559a1b9be4b0c3bfad3357a7?context=SH:SHW518173474:PL4327:1460619712763',
+        'only_matching': True,
+    }, {
+        'url': 'http://on.aol.com/video/519442220',
         'only_matching': True,
     }, {
         'url': 'aol-video:5707d6b8e4b090497b04f706',
         'only_matching': True,
-    }, {
-        'url': 'https://www.aol.com/video/playlist/PL8245/5ca79d19d21f1a04035db606/',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.aol.ca/video/view/u-s-woman-s-family-arrested-for-murder-first-pinned-on-panhandler-police/5c7ccf45bc03931fa04b2fe1/',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.aol.co.uk/video/view/-one-dead-and-22-hurt-in-bus-crash-/5cb3a6f3d21f1a072b457347/',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.aol.de/video/view/eva-braun-privataufnahmen-von-hitlers-geliebter-werden-digitalisiert/5cb2d49de98ab54c113d3d5d/',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.aol.jp/video/playlist/5a28e936a1334d000137da0c/5a28f3151e642219fde19831/',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -89,7 +73,7 @@ class AolIE(InfoExtractor):
 
         video_data = response['data']
         formats = []
-        m3u8_url = url_or_none(video_data.get('videoMasterPlaylist'))
+        m3u8_url = video_data.get('videoMasterPlaylist')
         if m3u8_url:
             formats.extend(self._extract_m3u8_formats(
                 m3u8_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
@@ -112,12 +96,6 @@ class AolIE(InfoExtractor):
                         'width': int(mobj.group(1)),
                         'height': int(mobj.group(2)),
                     })
-                else:
-                    qs = compat_parse_qs(compat_urllib_parse_urlparse(video_url).query)
-                    f.update({
-                        'width': int_or_none(qs.get('w', [None])[0]),
-                        'height': int_or_none(qs.get('h', [None])[0]),
-                    })
                 formats.append(f)
         self._sort_formats(formats, ('width', 'height', 'tbr', 'format_id'))
 

youtube_dl/extractor/beeg.py

@@ -9,8 +9,8 @@ from ..utils import (
 
 
 class BeegIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?beeg\.(?:com|porn(?:/video)?)/(?P<id>\d+)'
-    _TESTS = [{
+    _VALID_URL = r'https?://(?:www\.)?beeg\.com/(?P<id>\d+)'
+    _TEST = {
         'url': 'http://beeg.com/5416503',
         'md5': 'a1a1b1a8bc70a89e49ccfd113aed0820',
         'info_dict': {
@@ -24,13 +24,7 @@ class BeegIE(InfoExtractor):
             'tags': list,
             'age_limit': 18,
         }
-    }, {
-        'url': 'https://beeg.porn/video/5416503',
-        'only_matching': True,
-    }, {
-        'url': 'https://beeg.porn/5416503',
-        'only_matching': True,
-    }]
+    }
 
     def _real_extract(self, url):
         video_id = self._match_id(url)

youtube_dl/extractor/bfi.py

@@ -1,37 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-from ..utils import extract_attributes
-
-
-class BFIPlayerIE(InfoExtractor):
-    IE_NAME = 'bfi:player'
-    _VALID_URL = r'https?://player\.bfi\.org\.uk/[^/]+/film/watch-(?P<id>[\w-]+)-online'
-    _TEST = {
-        'url': 'https://player.bfi.org.uk/free/film/watch-computer-doctor-1974-online',
-        'md5': 'e8783ebd8e061ec4bc6e9501ed547de8',
-        'info_dict': {
-            'id': 'htNnhlZjE60C9VySkQEIBtU-cNV1Xx63',
-            'ext': 'mp4',
-            'title': 'Computer Doctor',
-            'description': 'md5:fb6c240d40c4dbe40428bdd62f78203b',
-        },
-        'skip': 'BFI Player films cannot be played outside of the UK',
-    }
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        entries = []
-        for player_el in re.findall(r'(?s)<[^>]+class="player"[^>]*>', webpage):
-            player_attr = extract_attributes(player_el)
-            ooyala_id = player_attr.get('data-video-id')
-            if not ooyala_id:
-                continue
-            entries.append(self.url_result(
-                'ooyala:' + ooyala_id, 'Ooyala',
-                ooyala_id, player_attr.get('data-label')))
-        return self.playlist_result(entries)

youtube_dl/extractor/biqle.py

@@ -2,96 +2,39 @@
 from __future__ import unicode_literals
 
 from .common import InfoExtractor
-from .vk import VKIE
-from ..utils import (
-    HEADRequest,
-    int_or_none,
-)
 
 
 class BIQLEIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?biqle\.(?:com|org|ru)/watch/(?P<id>-?\d+_\d+)'
     _TESTS = [{
-        # Youtube embed
-        'url': 'https://biqle.ru/watch/-115995369_456239081',
-        'md5': '97af5a06ee4c29bbf9c001bdb1cf5c06',
+        'url': 'http://www.biqle.ru/watch/847655_160197695',
+        'md5': 'ad5f746a874ccded7b8f211aeea96637',
         'info_dict': {
-            'id': '8v4f-avW-VI',
+            'id': '160197695',
             'ext': 'mp4',
-            'title': "PASSE-PARTOUT - L'ete c'est fait pour jouer",
-            'description': 'Passe-Partout',
-            'uploader_id': 'mrsimpsonstef3',
-            'uploader': 'Phanolito',
-            'upload_date': '20120822',
-        },
+            'title': 'Foo Fighters - The Pretender (Live at Wembley Stadium)',
+            'uploader': 'Andrey Rogozin',
+            'upload_date': '20110605',
+        }
     }, {
-        'url': 'http://biqle.org/watch/-44781847_168547604',
+        'url': 'https://biqle.org/watch/-44781847_168547604',
         'md5': '7f24e72af1db0edf7c1aaba513174f97',
         'info_dict': {
-            'id': '-44781847_168547604',
+            'id': '168547604',
             'ext': 'mp4',
             'title': 'Ребенок в шоке от автоматической мойки',
-            'timestamp': 1396633454,
             'uploader': 'Dmitry Kotov',
-            'upload_date': '20140404',
-            'uploader_id': '47850140',
         },
+        'skip': ' This video was marked as adult.  Embedding adult videos on external sites is prohibited.',
     }]
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
         webpage = self._download_webpage(url, video_id)
         embed_url = self._proto_relative_url(self._search_regex(
-            r'<iframe.+?src="((?:https?:)?//(?:daxab\.com|dxb\.to|[^/]+/player)/[^"]+)".*?></iframe>',
-            webpage, 'embed url'))
-        if VKIE.suitable(embed_url):
-            return self.url_result(embed_url, VKIE.ie_key(), video_id)
-
-        self._request_webpage(
-            HEADRequest(embed_url), video_id, headers={'Referer': url})
-        video_id, sig, _, access_token = self._get_cookies(embed_url)['video_ext'].value.split('%3A')
-        item = self._download_json(
-            'https://api.vk.com/method/video.get', video_id,
-            headers={'User-Agent': 'okhttp/3.4.1'}, query={
-                'access_token': access_token,
-                'sig': sig,
-                'v': 5.44,
-                'videos': video_id,
-            })['response']['items'][0]
-        title = item['title']
-
-        formats = []
-        for f_id, f_url in item.get('files', {}).items():
-            if f_id == 'external':
-                return self.url_result(f_url)
-            ext, height = f_id.split('_')
-            formats.append({
-                'format_id': height + 'p',
-                'url': f_url,
-                'height': int_or_none(height),
-                'ext': ext,
-            })
-        self._sort_formats(formats)
-
-        thumbnails = []
-        for k, v in item.items():
-            if k.startswith('photo_') and v:
-                width = k.replace('photo_', '')
-                thumbnails.append({
-                    'id': width,
-                    'url': v,
-                    'width': int_or_none(width),
-                })
+            r'<iframe.+?src="((?:http:)?//daxab\.com/[^"]+)".*?></iframe>', webpage, 'embed url'))
 
         return {
-            'id': video_id,
-            'title': title,
-            'formats': formats,
-            'comment_count': int_or_none(item.get('comments')),
-            'description': item.get('description'),
-            'duration': int_or_none(item.get('duration')),
-            'thumbnails': thumbnails,
-            'timestamp': int_or_none(item.get('date')),
-            'uploader': item.get('owner_id'),
-            'view_count': int_or_none(item.get('views')),
+            '_type': 'url_transparent',
+            'url': embed_url,
         }

youtube_dl/extractor/bitchute.py

@@ -55,11 +55,6 @@ class BitChuteIE(InfoExtractor):
         formats = [
             {'url': format_url}
             for format_url in orderedSet(format_urls)]
-
-        if not formats:
-            formats = self._parse_html5_media_entries(
-                url, webpage, video_id)[0]['formats']
-
         self._check_formats(formats, video_id)
         self._sort_formats(formats)
 
@@ -70,9 +65,8 @@ class BitChuteIE(InfoExtractor):
             webpage, default=None) or self._html_search_meta(
             'twitter:image:src', webpage, 'thumbnail')
         uploader = self._html_search_regex(
-            (r'(?s)<div class=["\']channel-banner.*?<p\b[^>]+\bclass=["\']name[^>]+>(.+?)</p>',
-             r'(?s)<p\b[^>]+\bclass=["\']video-author[^>]+>(.+?)</p>'),
-            webpage, 'uploader', fatal=False)
+            r'(?s)<p\b[^>]+\bclass=["\']video-author[^>]+>(.+?)</p>', webpage,
+            'uploader', fatal=False)
 
         return {
             'id': video_id,

youtube_dl/extractor/blinkx.py

@@ -32,8 +32,8 @@ class BlinkxIE(InfoExtractor):
         video_id = self._match_id(url)
         display_id = video_id[:8]
 
-        api_url = ('https://apib4.blinkx.com/api.php?action=play_video&'
-                   + 'video=%s' % video_id)
+        api_url = ('https://apib4.blinkx.com/api.php?action=play_video&' +
+                   'video=%s' % video_id)
         data_json = self._download_webpage(api_url, display_id)
         data = json.loads(data_json)['api']['results'][0]
         duration = None

youtube_dl/extractor/bravotv.py

@@ -1,8 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
-
 from .adobepass import AdobePassIE
 from ..utils import (
     smuggle_url,
@@ -14,16 +12,16 @@ from ..utils import (
 class BravoTVIE(AdobePassIE):
     _VALID_URL = r'https?://(?:www\.)?bravotv\.com/(?:[^/]+/)+(?P<id>[^/?#]+)'
     _TESTS = [{
-        'url': 'https://www.bravotv.com/top-chef/season-16/episode-15/videos/the-top-chef-season-16-winner-is',
-        'md5': 'e34684cfea2a96cd2ee1ef3a60909de9',
+        'url': 'http://www.bravotv.com/last-chance-kitchen/season-5/videos/lck-ep-12-fishy-finale',
+        'md5': '9086d0b7ef0ea2aabc4781d75f4e5863',
         'info_dict': {
-            'id': 'epL0pmK1kQlT',
+            'id': 'zHyk1_HU_mPy',
             'ext': 'mp4',
-            'title': 'The Top Chef Season 16 Winner Is...',
-            'description': 'Find out who takes the title of Top Chef!',
+            'title': 'LCK Ep 12: Fishy Finale',
+            'description': 'S13/E12: Two eliminated chefs have just 12 minutes to cook up a delicious fish dish.',
             'uploader': 'NBCU-BRAV',
-            'upload_date': '20190314',
-            'timestamp': 1552591860,
+            'upload_date': '20160302',
+            'timestamp': 1456945320,
         }
     }, {
         'url': 'http://www.bravotv.com/below-deck/season-3/ep-14-reunion-part-1',
@@ -34,38 +32,30 @@ class BravoTVIE(AdobePassIE):
         display_id = self._match_id(url)
         webpage = self._download_webpage(url, display_id)
         settings = self._parse_json(self._search_regex(
-            r'<script[^>]+data-drupal-selector="drupal-settings-json"[^>]*>({.+?})</script>', webpage, 'drupal settings'),
+            r'jQuery\.extend\(Drupal\.settings\s*,\s*({.+?})\);', webpage, 'drupal settings'),
             display_id)
         info = {}
         query = {
             'mbr': 'true',
         }
         account_pid, release_pid = [None] * 2
-        tve = settings.get('ls_tve')
+        tve = settings.get('sharedTVE')
         if tve:
             query['manifest'] = 'm3u'
-            mobj = re.search(r'<[^>]+id="pdk-player"[^>]+data-url=["\']?(?:https?:)?//player\.theplatform\.com/p/([^/]+)/(?:[^/]+/)*select/([^?#&"\']+)', webpage)
-            if mobj:
-                account_pid, tp_path = mobj.groups()
-                release_pid = tp_path.strip('/').split('/')[-1]
-            else:
-                account_pid = 'HNK2IC'
-                tp_path = release_pid = tve['release_pid']
+            account_pid = 'HNK2IC'
+            release_pid = tve['release_pid']
             if tve.get('entitlement') == 'auth':
-                adobe_pass = settings.get('tve_adobe_auth', {})
+                adobe_pass = settings.get('adobePass', {})
                 resource = self._get_mvpd_resource(
                     adobe_pass.get('adobePassResourceId', 'bravo'),
                     tve['title'], release_pid, tve.get('rating'))
                 query['auth'] = self._extract_mvpd_auth(
                     url, release_pid, adobe_pass.get('adobePassRequestorId', 'bravo'), resource)
         else:
-            shared_playlist = settings['ls_playlist']
+            shared_playlist = settings['shared_playlist']
             account_pid = shared_playlist['account_pid']
             metadata = shared_playlist['video_metadata'][shared_playlist['default_clip']]
-            tp_path = release_pid = metadata.get('release_pid')
-            if not release_pid:
-                release_pid = metadata['guid']
-                tp_path = 'media/guid/2140479951/' + release_pid
+            release_pid = metadata['release_pid']
             info.update({
                 'title': metadata['title'],
                 'description': metadata.get('description'),
@@ -77,7 +67,7 @@ class BravoTVIE(AdobePassIE):
             '_type': 'url_transparent',
             'id': release_pid,
             'url': smuggle_url(update_url_query(
-                'http://link.theplatform.com/s/%s/%s' % (account_pid, tp_path),
+                'http://link.theplatform.com/s/%s/%s' % (account_pid, release_pid),
                 query), {'force_smil_url': True}),
             'ie_key': 'ThePlatform',
         })

youtube_dl/extractor/byutv.py

@@ -3,13 +3,11 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from ..utils import parse_duration
 
 
 class BYUtvIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?byutv\.org/(?:watch|player)/(?!event/)(?P<id>[0-9a-f-]+)(?:/(?P<display_id>[^/?#&]+))?'
     _TESTS = [{
-        # ooyalaVOD
         'url': 'http://www.byutv.org/watch/6587b9a3-89d2-42a6-a7f7-fd2f81840a7d/studio-c-season-5-episode-5',
         'info_dict': {
             'id': 'ZvanRocTpW-G5_yZFeltTAMv6jxOU9KH',
@@ -24,20 +22,6 @@ class BYUtvIE(InfoExtractor):
             'skip_download': True,
         },
         'add_ie': ['Ooyala'],
-    }, {
-        # dvr
-        'url': 'https://www.byutv.org/player/8f1dab9b-b243-47c8-b525-3e2d021a3451/byu-softball-pacific-vs-byu-41219---game-2',
-        'info_dict': {
-            'id': '8f1dab9b-b243-47c8-b525-3e2d021a3451',
-            'display_id': 'byu-softball-pacific-vs-byu-41219---game-2',
-            'ext': 'mp4',
-            'title': 'Pacific vs. BYU (4/12/19)',
-            'description': 'md5:1ac7b57cb9a78015910a4834790ce1f3',
-            'duration': 11645,
-        },
-        'params': {
-            'skip_download': True
-        },
     }, {
         'url': 'http://www.byutv.org/watch/6587b9a3-89d2-42a6-a7f7-fd2f81840a7d',
         'only_matching': True,
@@ -51,42 +35,24 @@ class BYUtvIE(InfoExtractor):
         video_id = mobj.group('id')
         display_id = mobj.group('display_id') or video_id
 
-        info = self._download_json(
-            'https://api.byutv.org/api3/catalog/getvideosforcontent',
-            display_id, query={
+        ep = self._download_json(
+            'https://api.byutv.org/api3/catalog/getvideosforcontent', video_id,
+            query={
                 'contentid': video_id,
                 'channel': 'byutv',
                 'x-byutv-context': 'web$US',
             }, headers={
                 'x-byutv-context': 'web$US',
                 'x-byutv-platformkey': 'xsaaw9c7y5',
-            })
+            })['ooyalaVOD']
 
-        ep = info.get('ooyalaVOD')
-        if ep:
-            return {
-                '_type': 'url_transparent',
-                'ie_key': 'Ooyala',
-                'url': 'ooyala:%s' % ep['providerId'],
-                'id': video_id,
-                'display_id': display_id,
-                'title': ep.get('title'),
-                'description': ep.get('description'),
-                'thumbnail': ep.get('imageThumbnail'),
-            }
-
-        ep = info['dvr']
-        title = ep['title']
-        formats = self._extract_m3u8_formats(
-            ep['videoUrl'], video_id, 'mp4', entry_protocol='m3u8_native',
-            m3u8_id='hls')
-        self._sort_formats(formats)
         return {
+            '_type': 'url_transparent',
+            'ie_key': 'Ooyala',
+            'url': 'ooyala:%s' % ep['providerId'],
             'id': video_id,
             'display_id': display_id,
-            'title': title,
+            'title': ep.get('title'),
             'description': ep.get('description'),
             'thumbnail': ep.get('imageThumbnail'),
-            'duration': parse_duration(ep.get('length')),
-            'formats': formats,
         }

youtube_dl/extractor/canvas.py

@@ -17,7 +17,7 @@ from ..utils import (
 
 
 class CanvasIE(InfoExtractor):
-    _VALID_URL = r'https?://mediazone\.vrt\.be/api/v1/(?P<site_id>canvas|een|ketnet|vrt(?:video|nieuws)|sporza)/assets/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://mediazone\.vrt\.be/api/v1/(?P<site_id>canvas|een|ketnet|vrtvideo)/assets/(?P<id>[^/?#&]+)'
     _TESTS = [{
         'url': 'https://mediazone.vrt.be/api/v1/ketnet/assets/md-ast-4ac54990-ce66-4d00-a8ca-9eac86f4c475',
         'md5': '90139b746a0a9bd7bb631283f6e2a64e',
@@ -35,10 +35,6 @@ class CanvasIE(InfoExtractor):
         'url': 'https://mediazone.vrt.be/api/v1/canvas/assets/mz-ast-5e5f90b6-2d72-4c40-82c2-e134f884e93e',
         'only_matching': True,
     }]
-    _HLS_ENTRY_PROTOCOLS_MAP = {
-        'HLS': 'm3u8_native',
-        'HLS_AES': 'm3u8',
-    }
 
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
@@ -56,9 +52,9 @@ class CanvasIE(InfoExtractor):
             format_url, format_type = target.get('url'), target.get('type')
             if not format_url or not format_type:
                 continue
-            if format_type in self._HLS_ENTRY_PROTOCOLS_MAP:
+            if format_type == 'HLS':
                 formats.extend(self._extract_m3u8_formats(
-                    format_url, video_id, 'mp4', self._HLS_ENTRY_PROTOCOLS_MAP[format_type],
+                    format_url, video_id, 'mp4', entry_protocol='m3u8_native',
                     m3u8_id=format_type, fatal=False))
             elif format_type == 'HDS':
                 formats.extend(self._extract_f4m_formats(

youtube_dl/extractor/cbc.py

@@ -360,7 +360,7 @@ class CBCWatchVideoIE(CBCWatchBaseIE):
 
 class CBCWatchIE(CBCWatchBaseIE):
     IE_NAME = 'cbc.ca:watch'
-    _VALID_URL = r'https?://(?:gem|watch)\.cbc\.ca/(?:[^/]+/)+(?P<id>[0-9a-f-]+)'
+    _VALID_URL = r'https?://watch\.cbc\.ca/(?:[^/]+/)+(?P<id>[0-9a-f-]+)'
     _TESTS = [{
         # geo-restricted to Canada, bypassable
         'url': 'http://watch.cbc.ca/doc-zone/season-6/customer-disservice/38e815a-009e3ab12e4',
@@ -386,9 +386,6 @@ class CBCWatchIE(CBCWatchBaseIE):
             'description': 'Arthur, the sweetest 8-year-old aardvark, and his pals solve all kinds of problems with humour, kindness and teamwork.',
         },
         'playlist_mincount': 30,
-    }, {
-        'url': 'https://gem.cbc.ca/media/this-hour-has-22-minutes/season-26/episode-20/38e815a-0108c6c6a42',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):

youtube_dl/extractor/cbs.py

@@ -13,17 +13,13 @@ from ..utils import (
 
 class CBSBaseIE(ThePlatformFeedIE):
     def _parse_smil_subtitles(self, smil, namespace=None, subtitles_lang='en'):
-        subtitles = {}
-        for k, ext in [('sMPTE-TTCCURL', 'tt'), ('ClosedCaptionURL', 'ttml'), ('webVTTCaptionURL', 'vtt')]:
-            cc_e = find_xpath_attr(smil, self._xpath_ns('.//param', namespace), 'name', k)
-            if cc_e is not None:
-                cc_url = cc_e.get('value')
-                if cc_url:
-                    subtitles.setdefault(subtitles_lang, []).append({
-                        'ext': ext,
-                        'url': cc_url,
-                    })
-        return subtitles
+        closed_caption_e = find_xpath_attr(smil, self._xpath_ns('.//param', namespace), 'name', 'ClosedCaptionURL')
+        return {
+            'en': [{
+                'ext': 'ttml',
+                'url': closed_caption_e.attrib['value'],
+            }]
+        } if closed_caption_e is not None and closed_caption_e.attrib.get('value') else []
 
 
 class CBSIE(CBSBaseIE):
@@ -69,7 +65,7 @@ class CBSIE(CBSBaseIE):
         last_e = None
         for item in items_data.findall('.//item'):
             asset_type = xpath_text(item, 'assetType')
-            if not asset_type or asset_type in asset_types or 'HLS_FPS' in asset_type or 'DASH_CENC' in asset_type:
+            if not asset_type or asset_type in asset_types or asset_type in ('HLS_FPS', 'DASH_CENC'):
                 continue
             asset_types.append(asset_type)
             query = {

youtube_dl/extractor/cbsnews.py

@@ -1,62 +1,40 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
-import zlib
-
 from .common import InfoExtractor
 from .cbs import CBSIE
-from ..compat import (
-    compat_b64decode,
-    compat_urllib_parse_unquote,
-)
 from ..utils import (
     parse_duration,
 )
 
 
-class CBSNewsEmbedIE(CBSIE):
-    IE_NAME = 'cbsnews:embed'
-    _VALID_URL = r'https?://(?:www\.)?cbsnews\.com/embed/video[^#]*#(?P<id>.+)'
-    _TESTS = [{
-        'url': 'https://www.cbsnews.com/embed/video/?v=1.c9b5b61492913d6660db0b2f03579ef25e86307a#1Vb7b9s2EP5XBAHbT6Gt98PAMKTJ0se6LVjWYWtdGBR1stlIpEBSTtwi%2F%2FvuJNkNhmHdGxgM2NL57vjd6zt%2B8PngdN%2Fyg79qeGvhzN%2FLGrS%2F%2BuBLB531V28%2B%2BO7Qg7%2Fy97r2z3xZ42NW8yLhDbA0S0KWlHnIijwKWJBHZZnHBa8Cgbpdf%2F89NM9Hi9fXifhpr8sr%2FlP848tn%2BTdXycX25zh4cdX%2FvHl6PmmPqnWQv9w8Ed%2B9GjYRim07bFEqdG%2BZVHuwTm65A7bVRrYtR5lAyMox7pigF6W4k%2By91mjspGsJ%2BwVae4%2BsvdnaO1p73HkXs%2FVisUDTGm7R8IcdnOROeq%2B19qT1amhA1VJtPenoTUgrtfKc9m7Rq8dP7nnjwOB7wg7ADdNt7VX64DWAWlKhPtmDEq22g4GF99x6Dk9E8OSsankHXqPNKDxC%2FdK7MLKTircTDgsI3mmj4OBdSq64dy7fd1x577RU1rt4cvMtOaulFYOd%2FLewRWvDO9lIgXFpZSnkZmjbv5SxKTPoQXClFbpsf%2Fhbbpzs0IB3vb8KkyzJQ%2BywOAgCrMpgRrz%2BKk4fvb7kFbR4XJCu0gAdtNO7woCwZTu%2BBUs9bam%2Fds71drVerpeisgrubLjAB4nnOSkWQnfr5W6o1ku5Xpr1MgrCbL0M0vUyDtfLLK15WiYp47xKWSLyjFVpwVmVJSLIoCjSOFkv3W7oKsVliwZJcB9nwXpZ5GEQQwY8jNKqKCBrgjTLeFxgdCIpazojDgnRtn43J6kG7nZ6cAbxh0EeFFk4%2B1u867cY5u4344n%2FxXjCqAjucdTHgLKojNKmSfO8KRsOFY%2FzKEYCKEJBzv90QA9nfm9gL%2BHulaFqUkz9ULUYxl62B3U%2FRVNLA8IhggaPycOoBuwOCESciDQVSSUgiOMsROB%2FhKfwCKOzEk%2B4k6rWd4uuT%2FwTDz7K7t3d3WLO8ISD95jSPQbayBacthbz86XVgxHwhex5zawzgDOmtp%2F3GPcXn0VXHdSS029%2Fj99UC%2FwJUvyKQ%2FzKyixIEVlYJOn4RxxuaH43Ty9fbJ5OObykHH435XAzJTHeOF4hhEUXD8URe%2FQ%2FBT%2BMpf8d5GN02Ox%2FfiGsl7TA7POu1xZ5%2BbTzcAVKMe48mqcC21hkacVEVScM26liVVBnrKkC4CLKyzAvHu0lhEaTKMFwI3a4SN9MsrfYzdBLq2vkwRD1gVviLT8kY9h2CHH6Y%2Bix6609weFtey4ESp60WtyeWMy%2BsmBuhsoKIyuoT%2Bq2R%2FrW5qi3g%2FvzS2j40DoixDP8%2BKP0yUdpXJ4l6Vla%2Bg9vce%2BC4yM5YlUcbA%2F0jLKdpmTwvsdN5z88nAIe08%2F0HgxeG1iv%2B6Hlhjh7uiW0SDzYNI92L401uha3JKYk268UVRzdOzNQvAaJqoXzAc80dAV440NZ1WVVAAMRYQ2KrGJFmDUsq8saWSnjvIj8t78y%2FRa3JRnbHVfyFpfwoDiGpPgjzekyUiKNlU3OMlwuLMmzgvEojllYVE2Z1HhImvsnk%2BuhusTEoB21PAtSFodeFK3iYhXEH9WOG2%2FkOE833sfeG%2Ff5cfHtEFNXgYes0%2FXj7aGivUgJ9XpusCtoNcNYVVnJVrrDo0OmJAutHCpuZul4W9lLcfy7BnuLPT02%2ByXsCTk%2B9zhzswIN04YueNSK%2BPtM0jS88QdLqSLJDTLsuGZJNolm2yO0PXh3UPnz9Ix5bfIAqxPjvETQsDCEiPG4QbqNyhBZISxybLnZYCrW5H3Axp690%2F0BJdXtDZ5ITuM4xj3f4oUHGzc5JeJmZKpp%2FjwKh4wMV%2FV1yx3emLoR0MwbG4K%2F%2BZgVep3PnzXGDHZ6a3i%2Fk%2BJrONDN13%2Bnq6tBTYk4o7cLGhBtqCC4KwacGHpEVuoH5JNro%2FE6JfE6d5RydbiR76k%2BW5wioDHBIjw1euhHjUGRB0y5A97KoaPx6MlL%2BwgboUVtUFRI%2FLemgTpdtF59ii7pab08kuPcfWzs0l%2FRI5takWnFpka0zOgWRtYcuf9aIxZMxlwr6IiGpsb6j2DQUXPl%2FimXI599Ev7fWjoPD78A',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        item = self._parse_json(zlib.decompress(compat_b64decode(
-            compat_urllib_parse_unquote(self._match_id(url))),
-            -zlib.MAX_WBITS), None)['video']['items'][0]
-        return self._extract_video_info(item['mpxRefId'], 'cbsnews')
-
-
 class CBSNewsIE(CBSIE):
     IE_NAME = 'cbsnews'
     IE_DESC = 'CBS News'
-    _VALID_URL = r'https?://(?:www\.)?cbsnews\.com/(?:news|video)/(?P<id>[\da-z_-]+)'
+    _VALID_URL = r'https?://(?:www\.)?cbsnews\.com/(?:news|videos)/(?P<id>[\da-z_-]+)'
 
     _TESTS = [
         {
             # 60 minutes
             'url': 'http://www.cbsnews.com/news/artificial-intelligence-positioned-to-be-a-game-changer/',
             'info_dict': {
-                'id': 'Y_nf_aEg6WwO9OLAq0MpKaPgfnBUxfW4',
-                'ext': 'flv',
-                'title': 'Artificial Intelligence, real-life applications',
-                'description': 'md5:a7aaf27f1b4777244de8b0b442289304',
+                'id': '_B6Ga3VJrI4iQNKsir_cdFo9Re_YJHE_',
+                'ext': 'mp4',
+                'title': 'Artificial Intelligence',
+                'description': 'md5:8818145f9974431e0fb58a1b8d69613c',
                 'thumbnail': r're:^https?://.*\.jpg$',
-                'duration': 317,
+                'duration': 1606,
                 'uploader': 'CBSI-NEW',
-                'timestamp': 1476046464,
-                'upload_date': '20161009',
+                'timestamp': 1498431900,
+                'upload_date': '20170625',
             },
             'params': {
-                # rtmp download
+                # m3u8 download
                 'skip_download': True,
             },
         },
         {
-            'url': 'https://www.cbsnews.com/video/fort-hood-shooting-army-downplays-mental-illness-as-cause-of-attack/',
+            'url': 'http://www.cbsnews.com/videos/fort-hood-shooting-army-downplays-mental-illness-as-cause-of-attack/',
             'info_dict': {
                 'id': 'SNJBOYzXiWBOvaLsdzwH8fmtP1SCd91Y',
                 'ext': 'mp4',
@@ -82,29 +60,37 @@ class CBSNewsIE(CBSIE):
             # 48 hours
             'url': 'http://www.cbsnews.com/news/maria-ridulph-murder-will-the-nations-oldest-cold-case-to-go-to-trial-ever-get-solved/',
             'info_dict': {
+                'id': 'QpM5BJjBVEAUFi7ydR9LusS69DPLqPJ1',
+                'ext': 'mp4',
                 'title': 'Cold as Ice',
-                'description': 'Can a childhood memory solve the 1957 murder of 7-year-old Maria Ridulph?',
+                'description': 'Can a childhood memory of a friend\'s murder solve a 1957 cold case? "48 Hours" correspondent Erin Moriarty has the latest.',
+                'upload_date': '20170604',
+                'timestamp': 1496538000,
+                'uploader': 'CBSI-NEW',
+            },
+            'params': {
+                'skip_download': True,
             },
-            'playlist_mincount': 7,
         },
     ]
 
     def _real_extract(self, url):
-        display_id = self._match_id(url)
+        video_id = self._match_id(url)
 
-        webpage = self._download_webpage(url, display_id)
+        webpage = self._download_webpage(url, video_id)
 
-        entries = []
-        for embed_url in re.findall(r'<iframe[^>]+data-src="(https?://(?:www\.)?cbsnews\.com/embed/video/[^#]*#[^"]+)"', webpage):
-            entries.append(self.url_result(embed_url, CBSNewsEmbedIE.ie_key()))
-        if entries:
-            return self.playlist_result(
-                entries, playlist_title=self._html_search_meta(['og:title', 'twitter:title'], webpage),
-                playlist_description=self._html_search_meta(['og:description', 'twitter:description', 'description'], webpage))
+        video_info = self._parse_json(self._html_search_regex(
+            r'(?:<ul class="media-list items" id="media-related-items"[^>]*><li data-video-info|<div id="cbsNewsVideoPlayer" data-video-player-options)=\'({.+?})\'',
+            webpage, 'video JSON info', default='{}'), video_id, fatal=False)
+
+        if video_info:
+            item = video_info['item'] if 'item' in video_info else video_info
+        else:
+            state = self._parse_json(self._search_regex(
+                r'data-cbsvideoui-options=(["\'])(?P<json>{.+?})\1', webpage,
+                'playlist JSON info', group='json'), video_id)['state']
+            item = state['playlist'][state['pid']]
 
-        item = self._parse_json(self._html_search_regex(
-            r'CBSNEWS\.defaultPayload\s*=\s*({.+})',
-            webpage, 'video JSON info'), display_id)['items'][0]
         return self._extract_video_info(item['mpxRefId'], 'cbsnews')
 
 

youtube_dl/extractor/ccc.py

@@ -1,12 +1,9 @@
-# coding: utf-8
 from __future__ import unicode_literals
 
 from .common import InfoExtractor
 from ..utils import (
     int_or_none,
     parse_iso8601,
-    try_get,
-    url_or_none,
 )
 
 
@@ -21,13 +18,11 @@ class CCCIE(InfoExtractor):
             'id': '1839',
             'ext': 'mp4',
             'title': 'Introduction to Processor Design',
-            'creator': 'byterazor',
             'description': 'md5:df55f6d073d4ceae55aae6f2fd98a0ac',
             'thumbnail': r're:^https?://.*\.jpg$',
             'upload_date': '20131228',
             'timestamp': 1388188800,
             'duration': 3710,
-            'tags': list,
         }
     }, {
         'url': 'https://media.ccc.de/v/32c3-7368-shopshifting#download',
@@ -73,7 +68,6 @@ class CCCIE(InfoExtractor):
             'id': event_id,
             'display_id': display_id,
             'title': event_data['title'],
-            'creator': try_get(event_data, lambda x: ', '.join(x['persons'])),
             'description': event_data.get('description'),
             'thumbnail': event_data.get('thumb_url'),
             'timestamp': parse_iso8601(event_data.get('date')),
@@ -81,31 +75,3 @@ class CCCIE(InfoExtractor):
             'tags': event_data.get('tags'),
             'formats': formats,
         }
-
-
-class CCCPlaylistIE(InfoExtractor):
-    IE_NAME = 'media.ccc.de:lists'
-    _VALID_URL = r'https?://(?:www\.)?media\.ccc\.de/c/(?P<id>[^/?#&]+)'
-    _TESTS = [{
-        'url': 'https://media.ccc.de/c/30c3',
-        'info_dict': {
-            'title': '30C3',
-            'id': '30c3',
-        },
-        'playlist_count': 135,
-    }]
-
-    def _real_extract(self, url):
-        playlist_id = self._match_id(url).lower()
-
-        conf = self._download_json(
-            'https://media.ccc.de/public/conferences/' + playlist_id,
-            playlist_id)
-
-        entries = []
-        for e in conf['events']:
-            event_url = url_or_none(e.get('frontend_link'))
-            if event_url:
-                entries.append(self.url_result(event_url, ie=CCCIE.ie_key()))
-
-        return self.playlist_result(entries, playlist_id, conf.get('title'))

youtube_dl/extractor/cinemax.py

@@ -1,29 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .hbo import HBOBaseIE
-
-
-class CinemaxIE(HBOBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?cinemax\.com/(?P<path>[^/]+/video/[0-9a-z-]+-(?P<id>\d+))'
-    _TESTS = [{
-        'url': 'https://www.cinemax.com/warrior/video/s1-ep-1-recap-20126903',
-        'md5': '82e0734bba8aa7ef526c9dd00cf35a05',
-        'info_dict': {
-            'id': '20126903',
-            'ext': 'mp4',
-            'title': 'S1 Ep 1: Recap',
-        },
-        'expected_warnings': ['Unknown MIME type application/mp4 in DASH manifest'],
-    }, {
-        'url': 'https://www.cinemax.com/warrior/video/s1-ep-1-recap-20126903.embed',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        path, video_id = re.match(self._VALID_URL, url).groups()
-        info = self._extract_info('https://www.cinemax.com/%s.xml' % path, video_id)
-        info['id'] = video_id
-        return info

youtube_dl/extractor/ciscolive.py

@@ -65,8 +65,8 @@ class CiscoLiveBaseIE(InfoExtractor):
 
 
 class CiscoLiveSessionIE(CiscoLiveBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?ciscolive(?:\.cisco)?\.com/[^#]*#/session/(?P<id>[^/?&]+)'
-    _TESTS = [{
+    _VALID_URL = r'https?://ciscolive\.cisco\.com/on-demand-library/\??[^#]*#/session/(?P<id>[^/?&]+)'
+    _TEST = {
         'url': 'https://ciscolive.cisco.com/on-demand-library/?#/session/1423353499155001FoSs',
         'md5': 'c98acf395ed9c9f766941c70f5352e22',
         'info_dict': {
@@ -79,13 +79,7 @@ class CiscoLiveSessionIE(CiscoLiveBaseIE):
             'uploader_id': '5647924234001',
             'location': '16B Mezz.',
         },
-    }, {
-        'url': 'https://www.ciscolive.com/global/on-demand-library.html?search.event=ciscoliveemea2019#/session/15361595531500013WOU',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.ciscolive.com/global/on-demand-library.html?#/session/1490051371645001kNaS',
-        'only_matching': True,
-    }]
+    }
 
     def _real_extract(self, url):
         rf_id = self._match_id(url)
@@ -94,7 +88,7 @@ class CiscoLiveSessionIE(CiscoLiveBaseIE):
 
 
 class CiscoLiveSearchIE(CiscoLiveBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?ciscolive(?:\.cisco)?\.com/(?:global/)?on-demand-library(?:\.html|/)'
+    _VALID_URL = r'https?://ciscolive\.cisco\.com/on-demand-library/'
     _TESTS = [{
         'url': 'https://ciscolive.cisco.com/on-demand-library/?search.event=ciscoliveus2018&search.technicallevel=scpsSkillLevel_aintroductory&search.focus=scpsSessionFocus_designAndDeployment#/',
         'info_dict': {
@@ -104,9 +98,6 @@ class CiscoLiveSearchIE(CiscoLiveBaseIE):
     }, {
         'url': 'https://ciscolive.cisco.com/on-demand-library/?search.technology=scpsTechnology_applicationDevelopment&search.technology=scpsTechnology_ipv6&search.focus=scpsSessionFocus_troubleshootingTroubleshooting#/',
         'only_matching': True,
-    }, {
-        'url': 'https://www.ciscolive.com/global/on-demand-library.html?search.technicallevel=scpsSkillLevel_aintroductory&search.event=ciscoliveemea2019&search.technology=scpsTechnology_dataCenter&search.focus=scpsSessionFocus_bestPractices#/',
-        'only_matching': True,
     }]
 
     @classmethod

youtube_dl/extractor/cloudflarestream.py

@@ -10,8 +10,8 @@ class CloudflareStreamIE(InfoExtractor):
     _VALID_URL = r'''(?x)
                     https?://
                         (?:
-                            (?:watch\.)?(?:cloudflarestream\.com|videodelivery\.net)/|
-                            embed\.(?:cloudflarestream\.com|videodelivery\.net)/embed/[^/]+\.js\?.*?\bvideo=
+                            (?:watch\.)?cloudflarestream\.com/|
+                            embed\.cloudflarestream\.com/embed/[^/]+\.js\?.*?\bvideo=
                         )
                         (?P<id>[\da-f]+)
                     '''
@@ -31,9 +31,6 @@ class CloudflareStreamIE(InfoExtractor):
     }, {
         'url': 'https://cloudflarestream.com/31c9291ab41fac05471db4e73aa11717/manifest/video.mpd',
         'only_matching': True,
-    }, {
-        'url': 'https://embed.videodelivery.net/embed/r4xu.fla9.latest.js?video=81d80727f3022488598f68d323c1ad5e',
-        'only_matching': True,
     }]
 
     @staticmethod
@@ -41,7 +38,7 @@ class CloudflareStreamIE(InfoExtractor):
         return [
             mobj.group('url')
             for mobj in re.finditer(
-                r'<script[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//embed\.(?:cloudflarestream\.com|videodelivery\.net)/embed/[^/]+\.js\?.*?\bvideo=[\da-f]+?.*?)\1',
+                r'<script[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//embed\.cloudflarestream\.com/embed/[^/]+\.js\?.*?\bvideo=[\da-f]+?.*?)\1',
                 webpage)]
 
     def _real_extract(self, url):

youtube_dl/extractor/common.py

@@ -44,7 +44,6 @@ from ..utils import (
     compiled_regex_type,
     determine_ext,
     determine_protocol,
-    dict_get,
     error_to_compat_str,
     ExtractorError,
     extract_attributes,
@@ -57,17 +56,13 @@ from ..utils import (
     JSON_LD_RE,
     mimetype2ext,
     orderedSet,
-    parse_bitrate,
     parse_codecs,
     parse_duration,
     parse_iso8601,
     parse_m3u8_attributes,
-    parse_resolution,
     RegexNotFoundError,
     sanitized_Request,
     sanitize_filename,
-    str_or_none,
-    strip_or_none,
     unescapeHTML,
     unified_strdate,
     unified_timestamp,
@@ -113,13 +108,10 @@ class InfoExtractor(object):
                                    for RTMP - RTMP URL,
                                    for HLS - URL of the M3U8 media playlist,
                                    for HDS - URL of the F4M manifest,
-                                   for DASH
-                                     - HTTP URL to plain file media (in case of
-                                       unfragmented media)
-                                     - URL of the MPD manifest or base URL
-                                       representing the media if MPD manifest
-                                       is parsed from a string (in case of
-                                       fragmented media)
+                                   for DASH - URL of the MPD manifest or
+                                              base URL representing the media
+                                              if MPD manifest is parsed from
+                                              a string,
                                    for MSS - URL of the ISM manifest.
                     * manifest_url
                                  The URL of the manifest file in case of
@@ -543,11 +535,11 @@ class InfoExtractor(object):
             raise ExtractorError('An extractor error has occurred.', cause=e)
 
     def __maybe_fake_ip_and_retry(self, countries):
-        if (not self._downloader.params.get('geo_bypass_country', None)
-                and self._GEO_BYPASS
-                and self._downloader.params.get('geo_bypass', True)
-                and not self._x_forwarded_for_ip
-                and countries):
+        if (not self._downloader.params.get('geo_bypass_country', None) and
+                self._GEO_BYPASS and
+                self._downloader.params.get('geo_bypass', True) and
+                not self._x_forwarded_for_ip and
+                countries):
             country_code = random.choice(countries)
             self._x_forwarded_for_ip = GeoUtils.random_ipv4(country_code)
             if self._x_forwarded_for_ip:
@@ -683,8 +675,8 @@ class InfoExtractor(object):
 
     def __check_blocked(self, content):
         first_block = content[:512]
-        if ('<title>Access to this site is blocked</title>' in content
-                and 'Websense' in first_block):
+        if ('<title>Access to this site is blocked</title>' in content and
+                'Websense' in first_block):
             msg = 'Access to this webpage has been blocked by Websense filtering software in your network.'
             blocked_iframe = self._html_search_regex(
                 r'<iframe src="([^"]+)"', content,
@@ -702,8 +694,8 @@ class InfoExtractor(object):
             if block_msg:
                 msg += ' (Message: "%s")' % block_msg.replace('\n', ' ')
             raise ExtractorError(msg, expected=True)
-        if ('<title>TTK :: Доступ к ресурсу ограничен</title>' in content
-                and 'blocklist.rkn.gov.ru' in content):
+        if ('<title>TTK :: Доступ к ресурсу ограничен</title>' in content and
+                'blocklist.rkn.gov.ru' in content):
             raise ExtractorError(
                 'Access to this webpage has been blocked by decision of the Russian government. '
                 'Visit http://blocklist.rkn.gov.ru/ for a block reason.',
@@ -1710,8 +1702,8 @@ class InfoExtractor(object):
                 continue
             else:
                 tbr = float_or_none(
-                    last_stream_inf.get('AVERAGE-BANDWIDTH')
-                    or last_stream_inf.get('BANDWIDTH'), scale=1000)
+                    last_stream_inf.get('AVERAGE-BANDWIDTH') or
+                    last_stream_inf.get('BANDWIDTH'), scale=1000)
                 format_id = []
                 if m3u8_id:
                     format_id.append(m3u8_id)
@@ -2020,8 +2012,6 @@ class InfoExtractor(object):
         if res is False:
             return []
         mpd_doc, urlh = res
-        if mpd_doc is None:
-            return []
         mpd_base_url = base_url(urlh.geturl())
 
         return self._parse_mpd_formats(
@@ -2147,6 +2137,8 @@ class InfoExtractor(object):
                         bandwidth = int_or_none(representation_attrib.get('bandwidth'))
                         f = {
                             'format_id': '%s-%s' % (mpd_id, representation_id) if mpd_id else representation_id,
+                            # NB: mpd_url may be empty when MPD manifest is parsed from a string
+                            'url': mpd_url or base_url,
                             'manifest_url': mpd_url,
                             'ext': mimetype2ext(mime_type),
                             'width': int_or_none(representation_attrib.get('width')),
@@ -2285,14 +2277,10 @@ class InfoExtractor(object):
                                     fragment['duration'] = segment_duration
                                 fragments.append(fragment)
                             representation_ms_info['fragments'] = fragments
-                        # If there is a fragments key available then we correctly recognized fragmented media.
-                        # Otherwise we will assume unfragmented media with direct access. Technically, such
-                        # assumption is not necessarily correct since we may simply have no support for
-                        # some forms of fragmented media renditions yet, but for now we'll use this fallback.
+                        # NB: MPD manifest may contain direct URLs to unfragmented media.
+                        # No fragments key is present in this case.
                         if 'fragments' in representation_ms_info:
                             f.update({
-                                # NB: mpd_url may be empty when MPD manifest is parsed from a string
-                                'url': mpd_url or base_url,
                                 'fragment_base_url': base_url,
                                 'fragments': [],
                                 'protocol': 'http_dash_segments',
@@ -2303,10 +2291,6 @@ class InfoExtractor(object):
                                     f['url'] = initialization_url
                                 f['fragments'].append({location_key(initialization_url): initialization_url})
                             f['fragments'].extend(representation_ms_info['fragments'])
-                        else:
-                            # Assuming direct URL to unfragmented media.
-                            f['url'] = base_url
-
                         # According to [1, 5.3.5.2, Table 7, page 35] @id of Representation
                         # is not necessarily unique within a Period thus formats with
                         # the same `format_id` are quite possible. There are numerous examples
@@ -2481,50 +2465,25 @@ class InfoExtractor(object):
                 'subtitles': {},
             }
             media_attributes = extract_attributes(media_tag)
-            src = strip_or_none(media_attributes.get('src'))
+            src = media_attributes.get('src')
             if src:
                 _, formats = _media_formats(src, media_type)
                 media_info['formats'].extend(formats)
             media_info['thumbnail'] = absolute_url(media_attributes.get('poster'))
             if media_content:
                 for source_tag in re.findall(r'<source[^>]+>', media_content):
-                    s_attr = extract_attributes(source_tag)
-                    # data-video-src and data-src are non standard but seen
-                    # several times in the wild
-                    src = strip_or_none(dict_get(s_attr, ('src', 'data-video-src', 'data-src')))
+                    source_attributes = extract_attributes(source_tag)
+                    src = source_attributes.get('src')
                     if not src:
                         continue
-                    f = parse_content_type(s_attr.get('type'))
+                    f = parse_content_type(source_attributes.get('type'))
                     is_plain_url, formats = _media_formats(src, media_type, f)
                     if is_plain_url:
-                        # width, height, res, label and title attributes are
-                        # all not standard but seen several times in the wild
-                        labels = [
-                            s_attr.get(lbl)
-                            for lbl in ('label', 'title')
-                            if str_or_none(s_attr.get(lbl))
-                        ]
-                        width = int_or_none(s_attr.get('width'))
-                        height = (int_or_none(s_attr.get('height'))
-                                  or int_or_none(s_attr.get('res')))
-                        if not width or not height:
-                            for lbl in labels:
-                                resolution = parse_resolution(lbl)
-                                if not resolution:
-                                    continue
-                                width = width or resolution.get('width')
-                                height = height or resolution.get('height')
-                        for lbl in labels:
-                            tbr = parse_bitrate(lbl)
-                            if tbr:
-                                break
-                        else:
-                            tbr = None
+                        # res attribute is not standard but seen several times
+                        # in the wild
                         f.update({
-                            'width': width,
-                            'height': height,
-                            'tbr': tbr,
-                            'format_id': s_attr.get('label') or s_attr.get('title'),
+                            'height': int_or_none(source_attributes.get('res')),
+                            'format_id': source_attributes.get('label'),
                         })
                         f.update(formats[0])
                         media_info['formats'].append(f)
@@ -2534,7 +2493,7 @@ class InfoExtractor(object):
                     track_attributes = extract_attributes(track_tag)
                     kind = track_attributes.get('kind')
                     if not kind or kind in ('subtitles', 'captions'):
-                        src = strip_or_none(track_attributes.get('src'))
+                        src = track_attributes.get('src')
                         if not src:
                             continue
                         lang = track_attributes.get('srclang') or track_attributes.get('lang') or track_attributes.get('label')
@@ -2818,33 +2777,6 @@ class InfoExtractor(object):
         self._downloader.cookiejar.add_cookie_header(req)
         return compat_cookies.SimpleCookie(req.get_header('Cookie'))
 
-    def _apply_first_set_cookie_header(self, url_handle, cookie):
-        """
-        Apply first Set-Cookie header instead of the last. Experimental.
-
-        Some sites (e.g. [1-3]) may serve two cookies under the same name
-        in Set-Cookie header and expect the first (old) one to be set rather
-        than second (new). However, as of RFC6265 the newer one cookie
-        should be set into cookie store what actually happens.
-        We will workaround this issue by resetting the cookie to
-        the first one manually.
-        1. https://new.vk.com/
-        2. https://github.com/ytdl-org/youtube-dl/issues/9841#issuecomment-227871201
-        3. https://learning.oreilly.com/
-        """
-        for header, cookies in url_handle.headers.items():
-            if header.lower() != 'set-cookie':
-                continue
-            if sys.version_info[0] >= 3:
-                cookies = cookies.encode('iso-8859-1')
-            cookies = cookies.decode('utf-8')
-            cookie_value = re.search(
-                r'%s=(.+?);.*?\b[Dd]omain=(.+?)(?:[,;]|$)' % cookie, cookies)
-            if cookie_value:
-                value, domain = cookie_value.groups()
-                self._set_cookie(domain, cookie, value)
-                break
-
     def get_testcases(self, include_onlymatching=False):
         t = getattr(self, '_TEST', None)
         if t:
@@ -2875,8 +2807,8 @@ class InfoExtractor(object):
         return not any_restricted
 
     def extract_subtitles(self, *args, **kwargs):
-        if (self._downloader.params.get('writesubtitles', False)
-                or self._downloader.params.get('listsubtitles')):
+        if (self._downloader.params.get('writesubtitles', False) or
+                self._downloader.params.get('listsubtitles')):
             return self._get_subtitles(*args, **kwargs)
         return {}
 
@@ -2901,8 +2833,8 @@ class InfoExtractor(object):
         return ret
 
     def extract_automatic_captions(self, *args, **kwargs):
-        if (self._downloader.params.get('writeautomaticsub', False)
-                or self._downloader.params.get('listsubtitles')):
+        if (self._downloader.params.get('writeautomaticsub', False) or
+                self._downloader.params.get('listsubtitles')):
             return self._get_automatic_captions(*args, **kwargs)
         return {}
 
@@ -2910,9 +2842,9 @@ class InfoExtractor(object):
         raise NotImplementedError('This method must be implemented by subclasses')
 
     def mark_watched(self, *args, **kwargs):
-        if (self._downloader.params.get('mark_watched', False)
-                and (self._get_login_info()[0] is not None
-                     or self._downloader.params.get('cookiefile') is not None)):
+        if (self._downloader.params.get('mark_watched', False) and
+                (self._get_login_info()[0] is not None or
+                    self._downloader.params.get('cookiefile') is not None)):
             self._mark_watched(*args, **kwargs)
 
     def _mark_watched(self, *args, **kwargs):

youtube_dl/extractor/commonmistakes.py

@@ -32,19 +32,19 @@ class CommonMistakesIE(InfoExtractor):
 
 
 class UnicodeBOMIE(InfoExtractor):
-    IE_DESC = False
-    _VALID_URL = r'(?P<bom>\ufeff)(?P<id>.*)$'
+        IE_DESC = False
+        _VALID_URL = r'(?P<bom>\ufeff)(?P<id>.*)$'
 
-    # Disable test for python 3.2 since BOM is broken in re in this version
-    # (see https://github.com/ytdl-org/youtube-dl/issues/9751)
-    _TESTS = [] if (3, 0) < sys.version_info <= (3, 3) else [{
-        'url': '\ufeffhttp://www.youtube.com/watch?v=BaW_jenozKc',
-        'only_matching': True,
-    }]
+        # Disable test for python 3.2 since BOM is broken in re in this version
+        # (see https://github.com/ytdl-org/youtube-dl/issues/9751)
+        _TESTS = [] if (3, 0) < sys.version_info <= (3, 3) else [{
+            'url': '\ufeffhttp://www.youtube.com/watch?v=BaW_jenozKc',
+            'only_matching': True,
+        }]
 
-    def _real_extract(self, url):
-        real_url = self._match_id(url)
-        self.report_warning(
-            'Your URL starts with a Byte Order Mark (BOM). '
-            'Removing the BOM and looking for "%s" ...' % real_url)
-        return self.url_result(real_url)
+        def _real_extract(self, url):
+            real_url = self._match_id(url)
+            self.report_warning(
+                'Your URL starts with a Byte Order Mark (BOM). '
+                'Removing the BOM and looking for "%s" ...' % real_url)
+            return self.url_result(real_url)

youtube_dl/extractor/corus.py

@@ -13,9 +13,9 @@ class CorusIE(ThePlatformFeedIE):
                         (?:www\.)?
                         (?P<domain>
                             (?:globaltv|etcanada)\.com|
-                            (?:hgtv|foodnetwork|slice|history|showcase|bigbrothercanada)\.ca
+                            (?:hgtv|foodnetwork|slice|history|showcase)\.ca
                         )
-                        /(?:video/(?:[^/]+/)?|(?:[^/]+/)+(?:videos/[a-z0-9-]+-|video\.html\?.*?\bv=))
+                        /(?:video/|(?:[^/]+/)+(?:videos/[a-z0-9-]+-|video\.html\?.*?\bv=))
                         (?P<id>\d+)
                     '''
     _TESTS = [{
@@ -42,12 +42,6 @@ class CorusIE(ThePlatformFeedIE):
     }, {
         'url': 'http://www.showcase.ca/eyewitness/video/eyewitness++106/video.html?v=955070531919&p=1&s=da#video',
         'only_matching': True,
-    }, {
-        'url': 'http://www.bigbrothercanada.ca/video/1457812035894/',
-        'only_matching': True
-    }, {
-        'url': 'https://www.bigbrothercanada.ca/video/big-brother-canada-704/1457812035894/',
-        'only_matching': True
     }]
 
     _TP_FEEDS = {
@@ -79,10 +73,6 @@ class CorusIE(ThePlatformFeedIE):
             'feed_id': '9H6qyshBZU3E',
             'account_id': 2414426607,
         },
-        'bigbrothercanada': {
-            'feed_id': 'ChQqrem0lNUp',
-            'account_id': 2269680845,
-        },
     }
 
     def _real_extract(self, url):

youtube_dl/extractor/criterion.py

@@ -0,0 +1,39 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+
+class CriterionIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?criterion\.com/films/(?P<id>[0-9]+)-.+'
+    _TEST = {
+        'url': 'http://www.criterion.com/films/184-le-samourai',
+        'md5': 'bc51beba55685509883a9a7830919ec3',
+        'info_dict': {
+            'id': '184',
+            'ext': 'mp4',
+            'title': 'Le Samouraï',
+            'description': 'md5:a2b4b116326558149bef81f76dcbb93f',
+            'thumbnail': r're:^https?://.*\.jpg$',
+        }
+    }
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+
+        final_url = self._search_regex(
+            r'so\.addVariable\("videoURL", "(.+?)"\)\;', webpage, 'video url')
+        title = self._og_search_title(webpage)
+        description = self._html_search_meta('description', webpage)
+        thumbnail = self._search_regex(
+            r'so\.addVariable\("thumbnailURL", "(.+?)"\)\;',
+            webpage, 'thumbnail url')
+
+        return {
+            'id': video_id,
+            'url': final_url,
+            'title': title,
+            'description': description,
+            'thumbnail': thumbnail,
+        }

youtube_dl/extractor/cwtv.py

@@ -79,7 +79,7 @@ class CWTVIE(InfoExtractor):
         season = str_or_none(video_data.get('season'))
         episode = str_or_none(video_data.get('episode'))
         if episode and season:
-            episode = episode[len(season):]
+            episode = episode.lstrip(season)
 
         return {
             '_type': 'url_transparent',

youtube_dl/extractor/dailymail.py

@@ -45,8 +45,8 @@ class DailyMailIE(InfoExtractor):
         sources_url = (try_get(
             video_data,
             (lambda x: x['plugins']['sources']['url'],
-             lambda x: x['sources']['url']), compat_str)
-            or 'http://www.dailymail.co.uk/api/player/%s/video-sources.json' % video_id)
+             lambda x: x['sources']['url']), compat_str) or
+            'http://www.dailymail.co.uk/api/player/%s/video-sources.json' % video_id)
 
         video_sources = self._download_json(sources_url, video_id)
         body = video_sources.get('body')

youtube_dl/extractor/dctp.py

@@ -70,8 +70,8 @@ class DctpTvIE(InfoExtractor):
             endpoint = next(
                 server['endpoint']
                 for server in servers
-                if url_or_none(server.get('endpoint'))
-                and 'cloudfront' in server['endpoint'])
+                if url_or_none(server.get('endpoint')) and
+                'cloudfront' in server['endpoint'])
         else:
             endpoint = 'rtmpe://s2pqqn4u96e4j8.cloudfront.net/cfx/st/'
 

youtube_dl/extractor/dispeak.py

@@ -58,17 +58,10 @@ class DigitallySpeakingIE(InfoExtractor):
             stream_name = xpath_text(a_format, 'streamName', fatal=True)
             video_path = re.match(r'mp4\:(?P<path>.*)', stream_name).group('path')
             url = video_root + video_path
-            bitrate = xpath_text(a_format, 'bitrate')
-            tbr = int_or_none(bitrate)
-            vbr = int_or_none(self._search_regex(
-                r'-(\d+)\.mp4', video_path, 'vbr', default=None))
-            abr = tbr - vbr if tbr and vbr else None
+            vbr = xpath_text(a_format, 'bitrate')
             video_formats.append({
-                'format_id': bitrate,
                 'url': url,
-                'tbr': tbr,
-                'vbr': vbr,
-                'abr': abr,
+                'vbr': int_or_none(vbr),
             })
         return video_formats
 

youtube_dl/extractor/dramafever.py

@@ -0,0 +1,266 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import itertools
+import json
+
+from .common import InfoExtractor
+from ..compat import (
+    compat_HTTPError,
+    compat_urlparse,
+)
+from ..utils import (
+    clean_html,
+    ExtractorError,
+    int_or_none,
+    parse_age_limit,
+    parse_duration,
+    unified_timestamp,
+    url_or_none,
+)
+
+
+class DramaFeverBaseIE(InfoExtractor):
+    _NETRC_MACHINE = 'dramafever'
+
+    _CONSUMER_SECRET = 'DA59dtVXYLxajktV'
+
+    _consumer_secret = None
+
+    def _get_consumer_secret(self):
+        mainjs = self._download_webpage(
+            'http://www.dramafever.com/static/51afe95/df2014/scripts/main.js',
+            None, 'Downloading main.js', fatal=False)
+        if not mainjs:
+            return self._CONSUMER_SECRET
+        return self._search_regex(
+            r"var\s+cs\s*=\s*'([^']+)'", mainjs,
+            'consumer secret', default=self._CONSUMER_SECRET)
+
+    def _real_initialize(self):
+        self._consumer_secret = self._get_consumer_secret()
+        self._login()
+
+    def _login(self):
+        username, password = self._get_login_info()
+        if username is None:
+            return
+
+        login_form = {
+            'username': username,
+            'password': password,
+        }
+
+        try:
+            response = self._download_json(
+                'https://www.dramafever.com/api/users/login', None, 'Logging in',
+                data=json.dumps(login_form).encode('utf-8'), headers={
+                    'x-consumer-key': self._consumer_secret,
+                })
+        except ExtractorError as e:
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code in (403, 404):
+                response = self._parse_json(
+                    e.cause.read().decode('utf-8'), None)
+            else:
+                raise
+
+        # Successful login
+        if response.get('result') or response.get('guid') or response.get('user_guid'):
+            return
+
+        errors = response.get('errors')
+        if errors and isinstance(errors, list):
+            error = errors[0]
+            message = error.get('message') or error['reason']
+            raise ExtractorError('Unable to login: %s' % message, expected=True)
+        raise ExtractorError('Unable to log in')
+
+
+class DramaFeverIE(DramaFeverBaseIE):
+    IE_NAME = 'dramafever'
+    _VALID_URL = r'https?://(?:www\.)?dramafever\.com/(?:[^/]+/)?drama/(?P<id>[0-9]+/[0-9]+)(?:/|$)'
+    _TESTS = [{
+        'url': 'https://www.dramafever.com/drama/4274/1/Heirs/',
+        'info_dict': {
+            'id': '4274.1',
+            'ext': 'wvm',
+            'title': 'Heirs - Episode 1',
+            'description': 'md5:362a24ba18209f6276e032a651c50bc2',
+            'thumbnail': r're:^https?://.*\.jpg',
+            'duration': 3783,
+            'timestamp': 1381354993,
+            'upload_date': '20131009',
+            'series': 'Heirs',
+            'season_number': 1,
+            'episode': 'Episode 1',
+            'episode_number': 1,
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        },
+    }, {
+        'url': 'http://www.dramafever.com/drama/4826/4/Mnet_Asian_Music_Awards_2015/?ap=1',
+        'info_dict': {
+            'id': '4826.4',
+            'ext': 'flv',
+            'title': 'Mnet Asian Music Awards 2015',
+            'description': 'md5:3ff2ee8fedaef86e076791c909cf2e91',
+            'episode': 'Mnet Asian Music Awards 2015 - Part 3',
+            'episode_number': 4,
+            'thumbnail': r're:^https?://.*\.jpg',
+            'timestamp': 1450213200,
+            'upload_date': '20151215',
+            'duration': 5359,
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        },
+    }, {
+        'url': 'https://www.dramafever.com/zh-cn/drama/4972/15/Doctor_Romantic/',
+        'only_matching': True,
+    }]
+
+    def _call_api(self, path, video_id, note, fatal=False):
+        return self._download_json(
+            'https://www.dramafever.com/api/5/' + path,
+            video_id, note=note, headers={
+                'x-consumer-key': self._consumer_secret,
+            }, fatal=fatal)
+
+    def _get_subtitles(self, video_id):
+        subtitles = {}
+        subs = self._call_api(
+            'video/%s/subtitles/webvtt/' % video_id, video_id,
+            'Downloading subtitles JSON', fatal=False)
+        if not subs or not isinstance(subs, list):
+            return subtitles
+        for sub in subs:
+            if not isinstance(sub, dict):
+                continue
+            sub_url = url_or_none(sub.get('url'))
+            if not sub_url:
+                continue
+            subtitles.setdefault(
+                sub.get('code') or sub.get('language') or 'en', []).append({
+                    'url': sub_url
+                })
+        return subtitles
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url).replace('/', '.')
+
+        series_id, episode_number = video_id.split('.')
+
+        video = self._call_api(
+            'series/%s/episodes/%s/' % (series_id, episode_number), video_id,
+            'Downloading video JSON')
+
+        formats = []
+        download_assets = video.get('download_assets')
+        if download_assets and isinstance(download_assets, dict):
+            for format_id, format_dict in download_assets.items():
+                if not isinstance(format_dict, dict):
+                    continue
+                format_url = url_or_none(format_dict.get('url'))
+                if not format_url:
+                    continue
+                formats.append({
+                    'url': format_url,
+                    'format_id': format_id,
+                    'filesize': int_or_none(video.get('filesize')),
+                })
+
+        stream = self._call_api(
+            'video/%s/stream/' % video_id, video_id, 'Downloading stream JSON',
+            fatal=False)
+        if stream:
+            stream_url = stream.get('stream_url')
+            if stream_url:
+                formats.extend(self._extract_m3u8_formats(
+                    stream_url, video_id, 'mp4', entry_protocol='m3u8_native',
+                    m3u8_id='hls', fatal=False))
+        self._sort_formats(formats)
+
+        title = video.get('title') or 'Episode %s' % episode_number
+        description = video.get('description')
+        thumbnail = video.get('thumbnail')
+        timestamp = unified_timestamp(video.get('release_date'))
+        duration = parse_duration(video.get('duration'))
+        age_limit = parse_age_limit(video.get('tv_rating'))
+        series = video.get('series_title')
+        season_number = int_or_none(video.get('season'))
+
+        if series:
+            title = '%s - %s' % (series, title)
+
+        subtitles = self.extract_subtitles(video_id)
+
+        return {
+            'id': video_id,
+            'title': title,
+            'description': description,
+            'thumbnail': thumbnail,
+            'duration': duration,
+            'timestamp': timestamp,
+            'age_limit': age_limit,
+            'series': series,
+            'season_number': season_number,
+            'episode_number': int_or_none(episode_number),
+            'formats': formats,
+            'subtitles': subtitles,
+        }
+
+
+class DramaFeverSeriesIE(DramaFeverBaseIE):
+    IE_NAME = 'dramafever:series'
+    _VALID_URL = r'https?://(?:www\.)?dramafever\.com/(?:[^/]+/)?drama/(?P<id>[0-9]+)(?:/(?:(?!\d+(?:/|$)).+)?)?$'
+    _TESTS = [{
+        'url': 'http://www.dramafever.com/drama/4512/Cooking_with_Shin/',
+        'info_dict': {
+            'id': '4512',
+            'title': 'Cooking with Shin',
+            'description': 'md5:84a3f26e3cdc3fb7f500211b3593b5c1',
+        },
+        'playlist_count': 4,
+    }, {
+        'url': 'http://www.dramafever.com/drama/124/IRIS/',
+        'info_dict': {
+            'id': '124',
+            'title': 'IRIS',
+            'description': 'md5:b3a30e587cf20c59bd1c01ec0ee1b862',
+        },
+        'playlist_count': 20,
+    }]
+
+    _PAGE_SIZE = 60  # max is 60 (see http://api.drama9.com/#get--api-4-episode-series-)
+
+    def _real_extract(self, url):
+        series_id = self._match_id(url)
+
+        series = self._download_json(
+            'http://www.dramafever.com/api/4/series/query/?cs=%s&series_id=%s'
+            % (self._consumer_secret, series_id),
+            series_id, 'Downloading series JSON')['series'][series_id]
+
+        title = clean_html(series['name'])
+        description = clean_html(series.get('description') or series.get('description_short'))
+
+        entries = []
+        for page_num in itertools.count(1):
+            episodes = self._download_json(
+                'http://www.dramafever.com/api/4/episode/series/?cs=%s&series_id=%s&page_size=%d&page_number=%d'
+                % (self._consumer_secret, series_id, self._PAGE_SIZE, page_num),
+                series_id, 'Downloading episodes JSON page #%d' % page_num)
+            for episode in episodes.get('value', []):
+                episode_url = episode.get('episode_url')
+                if not episode_url:
+                    continue
+                entries.append(self.url_result(
+                    compat_urlparse.urljoin(url, episode_url),
+                    'DramaFever', episode.get('guid')))
+            if page_num == episodes['num_pages']:
+                break
+
+        return self.playlist_result(entries, series_id, title, description)

youtube_dl/extractor/dvtv.py

@@ -10,16 +10,16 @@ from ..utils import (
     int_or_none,
     js_to_json,
     mimetype2ext,
-    try_get,
     unescapeHTML,
-    parse_iso8601,
 )
 
 
 class DVTVIE(InfoExtractor):
     IE_NAME = 'dvtv'
     IE_DESC = 'http://video.aktualne.cz/'
+
     _VALID_URL = r'https?://video\.aktualne\.cz/(?:[^/]+/)+r~(?P<id>[0-9a-f]{32})'
+
     _TESTS = [{
         'url': 'http://video.aktualne.cz/dvtv/vondra-o-ceskem-stoleti-pri-pohledu-na-havla-mi-bylo-trapne/r~e5efe9ca855511e4833a0025900fea04/',
         'md5': '67cb83e4a955d36e1b5d31993134a0c2',
@@ -28,13 +28,11 @@ class DVTVIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'Vondra o Českém století: Při pohledu na Havla mi bylo trapně',
             'duration': 1484,
-            'upload_date': '20141217',
-            'timestamp': 1418792400,
         }
     }, {
         'url': 'http://video.aktualne.cz/dvtv/dvtv-16-12-2014-utok-talibanu-boj-o-kliniku-uprchlici/r~973eb3bc854e11e498be002590604f2e/',
         'info_dict': {
-            'title': r'DVTV 16. 12. 2014: útok Talibanu, boj o kliniku, uprchlíci',
+            'title': r're:^DVTV 16\. 12\. 2014: útok Talibanu, boj o kliniku, uprchlíci',
             'id': '973eb3bc854e11e498be002590604f2e',
         },
         'playlist': [{
@@ -86,8 +84,6 @@ class DVTVIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'Zeman si jen léčí mindráky, Sobotku nenávidí a Babiš se mu teď hodí, tvrdí Kmenta',
             'duration': 1103,
-            'upload_date': '20170511',
-            'timestamp': 1494514200,
         },
         'params': {
             'skip_download': True,
@@ -95,59 +91,43 @@ class DVTVIE(InfoExtractor):
     }, {
         'url': 'http://video.aktualne.cz/v-cechach-poprve-zazni-zelenkova-zrestaurovana-mse/r~45b4b00483ec11e4883b002590604f2e/',
         'only_matching': True,
-    }, {
-        # Test live stream video (liveStarter) parsing
-        'url': 'https://video.aktualne.cz/dvtv/zive-mistryne-sveta-eva-samkova-po-navratu-ze-sampionatu/r~182654c2288811e990fd0cc47ab5f122/',
-        'md5': '2e552e483f2414851ca50467054f9d5d',
-        'info_dict': {
-            'id': '8d116360288011e98c840cc47ab5f122',
-            'ext': 'mp4',
-            'title': 'Živě: Mistryně světa Eva Samková po návratu ze šampionátu',
-            'upload_date': '20190204',
-            'timestamp': 1549289591,
-        },
-        'params': {
-            # Video content is no longer available
-            'skip_download': True,
-        },
     }]
 
-    def _parse_video_metadata(self, js, video_id, timestamp):
+    def _parse_video_metadata(self, js, video_id, live_js=None):
         data = self._parse_json(js, video_id, transform_source=js_to_json)
+        if live_js:
+            data.update(self._parse_json(
+                live_js, video_id, transform_source=js_to_json))
+
         title = unescapeHTML(data['title'])
 
-        live_starter = try_get(data, lambda x: x['plugins']['liveStarter'], dict)
-        if live_starter:
-            data.update(live_starter)
-
         formats = []
-        for tracks in data.get('tracks', {}).values():
-            for video in tracks:
-                video_url = video.get('src')
-                if not video_url:
-                    continue
-                video_type = video.get('type')
-                ext = determine_ext(video_url, mimetype2ext(video_type))
-                if video_type == 'application/vnd.apple.mpegurl' or ext == 'm3u8':
-                    formats.extend(self._extract_m3u8_formats(
-                        video_url, video_id, 'mp4', entry_protocol='m3u8_native',
-                        m3u8_id='hls', fatal=False))
-                elif video_type == 'application/dash+xml' or ext == 'mpd':
-                    formats.extend(self._extract_mpd_formats(
-                        video_url, video_id, mpd_id='dash', fatal=False))
-                else:
-                    label = video.get('label')
-                    height = self._search_regex(
-                        r'^(\d+)[pP]', label or '', 'height', default=None)
-                    format_id = ['http']
-                    for f in (ext, label):
-                        if f:
-                            format_id.append(f)
-                    formats.append({
-                        'url': video_url,
-                        'format_id': '-'.join(format_id),
-                        'height': int_or_none(height),
-                    })
+        for video in data['sources']:
+            video_url = video.get('file')
+            if not video_url:
+                continue
+            video_type = video.get('type')
+            ext = determine_ext(video_url, mimetype2ext(video_type))
+            if video_type == 'application/vnd.apple.mpegurl' or ext == 'm3u8':
+                formats.extend(self._extract_m3u8_formats(
+                    video_url, video_id, 'mp4', entry_protocol='m3u8_native',
+                    m3u8_id='hls', fatal=False))
+            elif video_type == 'application/dash+xml' or ext == 'mpd':
+                formats.extend(self._extract_mpd_formats(
+                    video_url, video_id, mpd_id='dash', fatal=False))
+            else:
+                label = video.get('label')
+                height = self._search_regex(
+                    r'^(\d+)[pP]', label or '', 'height', default=None)
+                format_id = ['http']
+                for f in (ext, label):
+                    if f:
+                        format_id.append(f)
+                formats.append({
+                    'url': video_url,
+                    'format_id': '-'.join(format_id),
+                    'height': int_or_none(height),
+                })
         self._sort_formats(formats)
 
         return {
@@ -156,29 +136,41 @@ class DVTVIE(InfoExtractor):
             'description': data.get('description'),
             'thumbnail': data.get('image'),
             'duration': int_or_none(data.get('duration')),
-            'timestamp': int_or_none(timestamp),
+            'timestamp': int_or_none(data.get('pubtime')),
             'formats': formats
         }
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
+
         webpage = self._download_webpage(url, video_id)
-        timestamp = parse_iso8601(self._html_search_meta(
-            'article:published_time', webpage, 'published time', default=None))
 
-        items = re.findall(r'(?s)playlist\.push\(({.+?})\);', webpage)
-        if items:
-            return self.playlist_result(
-                [self._parse_video_metadata(i, video_id, timestamp) for i in items],
-                video_id, self._html_search_meta('twitter:title', webpage))
-
-        item = self._search_regex(
-            r'(?s)BBXPlayer\.setup\((.+?)\);',
+        # live content
+        live_item = self._search_regex(
+            r'(?s)embedData[0-9a-f]{32}\.asset\.liveStarter\s*=\s*(\{.+?\});',
             webpage, 'video', default=None)
+
+        # single video
+        item = self._search_regex(
+            r'(?s)embedData[0-9a-f]{32}\[["\']asset["\']\]\s*=\s*(\{.+?\});',
+            webpage, 'video', default=None)
+
         if item:
-            # remove function calls (ex. htmldeentitize)
-            # TODO this should be fixed in a general way in the js_to_json
-            item = re.sub(r'\w+?\((.+)\)', r'\1', item)
-            return self._parse_video_metadata(item, video_id, timestamp)
+            return self._parse_video_metadata(item, video_id, live_item)
+
+        # playlist
+        items = re.findall(
+            r"(?s)BBX\.context\.assets\['[0-9a-f]{32}'\]\.push\(({.+?})\);",
+            webpage)
+        if not items:
+            items = re.findall(r'(?s)var\s+asset\s*=\s*({.+?});\n', webpage)
+
+        if items:
+            return {
+                '_type': 'playlist',
+                'id': video_id,
+                'title': self._og_search_title(webpage),
+                'entries': [self._parse_video_metadata(i, video_id) for i in items]
+            }
 
         raise ExtractorError('Could not find neither video nor playlist')

youtube_dl/extractor/escapist.py

@@ -1,11 +1,14 @@
 from __future__ import unicode_literals
 
+import json
+
 from .common import InfoExtractor
 from ..utils import (
     determine_ext,
     clean_html,
     int_or_none,
     float_or_none,
+    sanitized_Request,
 )
 
 
@@ -33,7 +36,7 @@ def _decrypt_config(key, string):
 
 
 class EscapistIE(InfoExtractor):
-    _VALID_URL = r'https?://?(?:(?:www|v1)\.)?escapistmagazine\.com/videos/view/[^/]+/(?P<id>[0-9]+)'
+    _VALID_URL = r'https?://?(?:www\.)?escapistmagazine\.com/videos/view/[^/?#]+/(?P<id>[0-9]+)-[^/?#]*(?:$|[?#])'
     _TESTS = [{
         'url': 'http://www.escapistmagazine.com/videos/view/the-escapist-presents/6618-Breaking-Down-Baldurs-Gate',
         'md5': 'ab3a706c681efca53f0a35f1415cf0d1',
@@ -58,12 +61,6 @@ class EscapistIE(InfoExtractor):
             'duration': 304,
             'uploader': 'The Escapist',
         }
-    }, {
-        'url': 'http://escapistmagazine.com/videos/view/the-escapist-presents/6618',
-        'only_matching': True,
-    }, {
-        'url': 'https://v1.escapistmagazine.com/videos/view/the-escapist-presents/6618-Breaking-Down-Baldurs-Gate',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -77,20 +74,19 @@ class EscapistIE(InfoExtractor):
         video_id = ims_video['videoID']
         key = ims_video['hash']
 
-        config = self._download_webpage(
-            'http://www.escapistmagazine.com/videos/vidconfig.php',
-            video_id, 'Downloading video config', headers={
-                'Referer': url,
-            }, query={
-                'videoID': video_id,
-                'hash': key,
-            })
+        config_req = sanitized_Request(
+            'http://www.escapistmagazine.com/videos/'
+            'vidconfig.php?videoID=%s&hash=%s' % (video_id, key))
+        config_req.add_header('Referer', url)
+        config = self._download_webpage(config_req, video_id, 'Downloading video config')
 
-        data = self._parse_json(_decrypt_config(key, config), video_id)
+        data = json.loads(_decrypt_config(key, config))
 
         video_data = data['videoData']
 
         title = clean_html(video_data['title'])
+        duration = float_or_none(video_data.get('duration'), 1000)
+        uploader = video_data.get('publisher')
 
         formats = [{
             'url': video['src'],
@@ -103,9 +99,8 @@ class EscapistIE(InfoExtractor):
             'id': video_id,
             'formats': formats,
             'title': title,
-            'thumbnail': self._og_search_thumbnail(webpage) or data.get('poster'),
+            'thumbnail': self._og_search_thumbnail(webpage),
             'description': self._og_search_description(webpage),
-            'duration': float_or_none(video_data.get('duration'), 1000),
-            'uploader': video_data.get('publisher'),
-            'series': video_data.get('show'),
+            'duration': duration,
+            'uploader': uploader,
         }

youtube_dl/extractor/expressen.py

@@ -82,8 +82,8 @@ class ExpressenIE(InfoExtractor):
         title = info.get('titleRaw') or data['title']
         description = info.get('descriptionRaw')
         thumbnail = info.get('socialMediaImage') or data.get('image')
-        duration = int_or_none(info.get('videoTotalSecondsDuration')
-                               or data.get('totalSecondsDuration'))
+        duration = int_or_none(info.get('videoTotalSecondsDuration') or
+                               data.get('totalSecondsDuration'))
         timestamp = unified_timestamp(info.get('publishDate'))
 
         return {

youtube_dl/extractor/extractors.py

@@ -20,7 +20,6 @@ from .acast import (
 )
 from .addanime import AddAnimeIE
 from .adn import ADNIE
-from .adobeconnect import AdobeConnectIE
 from .adobetv import (
     AdobeTVIE,
     AdobeTVShowIE,
@@ -39,7 +38,9 @@ from .alphaporno import AlphaPornoIE
 from .amcnetworks import AMCNetworksIE
 from .americastestkitchen import AmericasTestKitchenIE
 from .animeondemand import AnimeOnDemandIE
+from .anitube import AnitubeIE
 from .anvato import AnvatoIE
+from .anysex import AnySexIE
 from .aol import AolIE
 from .allocine import AllocineIE
 from .aliexpress import AliExpressLiveIE
@@ -107,7 +108,6 @@ from .behindkink import BehindKinkIE
 from .bellmedia import BellMediaIE
 from .beatport import BeatportIE
 from .bet import BetIE
-from .bfi import BFIPlayerIE
 from .bigflix import BigflixIE
 from .bild import BildIE
 from .bilibili import (
@@ -173,15 +173,11 @@ from .cbs import CBSIE
 from .cbslocal import CBSLocalIE
 from .cbsinteractive import CBSInteractiveIE
 from .cbsnews import (
-    CBSNewsEmbedIE,
     CBSNewsIE,
     CBSNewsLiveVideoIE,
 )
 from .cbssports import CBSSportsIE
-from .ccc import (
-    CCCIE,
-    CCCPlaylistIE,
-)
+from .ccc import CCCIE
 from .ccma import CCMAIE
 from .cctv import CCTVIE
 from .cda import CDAIE
@@ -198,7 +194,6 @@ from .chirbit import (
     ChirbitProfileIE,
 )
 from .cinchcast import CinchcastIE
-from .cinemax import CinemaxIE
 from .ciscolive import (
     CiscoLiveSessionIE,
     CiscoLiveSearchIE,
@@ -241,6 +236,7 @@ from .condenast import CondeNastIE
 from .corus import CorusIE
 from .cracked import CrackedIE
 from .crackle import CrackleIE
+from .criterion import CriterionIE
 from .crooksandliars import CrooksAndLiarsIE
 from .crunchyroll import (
     CrunchyrollIE,
@@ -287,6 +283,10 @@ from .dplay import (
     DPlayIE,
     DPlayItIE,
 )
+from .dramafever import (
+    DramaFeverIE,
+    DramaFeverSeriesIE,
+)
 from .dreisat import DreiSatIE
 from .drbonanza import DRBonanzaIE
 from .drtuber import DrTuberIE
@@ -442,7 +442,10 @@ from .goshgay import GoshgayIE
 from .gputechconf import GPUTechConfIE
 from .groupon import GrouponIE
 from .hark import HarkIE
-from .hbo import HBOIE
+from .hbo import (
+    HBOIE,
+    HBOEpisodeIE,
+)
 from .hearthisat import HearThisAtIE
 from .heise import HeiseIE
 from .hellporno import HellPornoIE
@@ -631,11 +634,7 @@ from .massengeschmacktv import MassengeschmackTVIE
 from .matchtv import MatchTVIE
 from .mdr import MDRIE
 from .mediaset import MediasetIE
-from .mediasite import (
-    MediasiteIE,
-    MediasiteCatalogIE,
-    MediasiteNamedCatalogIE,
-)
+from .mediasite import MediasiteIE
 from .medici import MediciIE
 from .megaphone import MegaphoneIE
 from .meipai import MeipaiIE
@@ -772,6 +771,13 @@ from .nova import (
     NovaEmbedIE,
     NovaIE,
 )
+from .novamov import (
+    AuroraVidIE,
+    CloudTimeIE,
+    NowVideoIE,
+    VideoWeedIE,
+    WholeCloudIE,
+)
 from .nowness import (
     NownessIE,
     NownessPlaylistIE,
@@ -801,8 +807,6 @@ from .nrk import (
     NRKTVSeasonIE,
     NRKTVSeriesIE,
 )
-from .nrl import NRLTVIE
-from .ntvcojp import NTVCoJpCUIE
 from .ntvde import NTVDeIE
 from .ntvru import NTVRuIE
 from .nytimes import (
@@ -826,10 +830,7 @@ from .ooyala import (
     OoyalaIE,
     OoyalaExternalIE,
 )
-from .openload import (
-    OpenloadIE,
-    VerystreamIE,
-)
+from .openload import OpenloadIE
 from .ora import OraTVIE
 from .orf import (
     ORFTVthekIE,
@@ -866,10 +867,6 @@ from .picarto import (
 from .piksel import PikselIE
 from .pinkbike import PinkbikeIE
 from .pladform import PladformIE
-from .platzi import (
-    PlatziIE,
-    PlatziCourseIE,
-)
 from .playfm import PlayFMIE
 from .playplustv import PlayPlusTVIE
 from .plays import PlaysTVIE
@@ -889,6 +886,7 @@ from .polskieradio import (
 from .popcorntv import PopcornTVIE
 from .porn91 import Porn91IE
 from .porncom import PornComIE
+from .pornflip import PornFlipIE
 from .pornhd import PornHdIE
 from .pornhub import (
     PornHubIE,
@@ -903,6 +901,7 @@ from .puhutv import (
     PuhuTVSerieIE,
 )
 from .presstv import PressTVIE
+from .primesharetv import PrimeShareTVIE
 from .promptfile import PromptFileIE
 from .prosiebensat1 import ProSiebenSat1IE
 from .puls4 import Puls4IE
@@ -938,10 +937,7 @@ from .raywenderlich import (
 )
 from .rbmaradio import RBMARadioIE
 from .rds import RDSIE
-from .redbulltv import (
-    RedBullTVIE,
-    RedBullTVRrnContentIE,
-)
+from .redbulltv import RedBullTVIE
 from .reddit import (
     RedditIE,
     RedditRIE,
@@ -982,6 +978,7 @@ from .rtvnh import RTVNHIE
 from .rtvs import RTVSIE
 from .rudo import RudoIE
 from .ruhd import RUHDIE
+from .ruleporn import RulePornIE
 from .rutube import (
     RutubeIE,
     RutubeChannelIE,
@@ -1031,10 +1028,7 @@ from .skynewsarabia import (
     SkyNewsArabiaIE,
     SkyNewsArabiaArticleIE,
 )
-from .sky import (
-    SkyNewsIE,
-    SkySportsIE,
-)
+from .skysports import SkySportsIE
 from .slideshare import SlideshareIE
 from .slideslive import SlidesLiveIE
 from .slutload import SlutloadIE
@@ -1096,12 +1090,7 @@ from .streamcloud import StreamcloudIE
 from .streamcz import StreamCZIE
 from .streetvoice import StreetVoiceIE
 from .stretchinternet import StretchInternetIE
-from .stv import STVPlayerIE
 from .sunporno import SunPornoIE
-from .sverigesradio import (
-    SverigesRadioEpisodeIE,
-    SverigesRadioPublicationIE,
-)
 from .svt import (
     SVTIE,
     SVTPageIE,
@@ -1129,7 +1118,6 @@ from .teachertube import (
 )
 from .teachingchannel import TeachingChannelIE
 from .teamcoco import TeamcocoIE
-from .teamtreehouse import TeamTreeHouseIE
 from .techtalks import TechTalksIE
 from .ted import TEDIE
 from .tele5 import Tele5IE
@@ -1311,6 +1299,7 @@ from .viddler import ViddlerIE
 from .videa import VideaIE
 from .videodetective import VideoDetectiveIE
 from .videofyme import VideofyMeIE
+from .videomega import VideoMegaIE
 from .videomore import (
     VideomoreIE,
     VideomoreVideoIE,
@@ -1416,13 +1405,17 @@ from .webofstories import (
     WebOfStoriesPlaylistIE,
 )
 from .weibo import (
-    WeiboIE,
+    WeiboIE, 
     WeiboMobileIE
 )
 from .weiqitv import WeiqiTVIE
 from .wimp import WimpIE
 from .wistia import WistiaIE
 from .worldstarhiphop import WorldStarHipHopIE
+from .wrzuta import (
+    WrzutaIE,
+    WrzutaPlaylistIE,
+)
 from .wsj import (
     WSJIE,
     WSJArticleIE,
@@ -1455,16 +1448,13 @@ from .xxxymovies import XXXYMoviesIE
 from .yahoo import (
     YahooIE,
     YahooSearchIE,
-    YahooGyaOPlayerIE,
-    YahooGyaOIE,
 )
-from .yandexdisk import YandexDiskIE
 from .yandexmusic import (
     YandexMusicTrackIE,
     YandexMusicAlbumIE,
     YandexMusicPlaylistIE,
 )
-from .yandexvideo import YandexVideoIE
+from .yandexdisk import YandexDiskIE
 from .yapfiles import YapFilesIE
 from .yesjapan import YesJapanIE
 from .yinyuetai import YinYueTaiIE

youtube_dl/extractor/fourtube.py

@@ -22,6 +22,8 @@ from ..utils import (
 
 
 class FourTubeBaseIE(InfoExtractor):
+    _TKN_HOST = 'tkn.kodicdn.com'
+
     def _extract_formats(self, url, video_id, media_id, sources):
         token_url = 'https://%s/%s/desktop/%s' % (
             self._TKN_HOST, media_id, '+'.join(sources))
@@ -118,7 +120,6 @@ class FourTubeIE(FourTubeBaseIE):
     IE_NAME = '4tube'
     _VALID_URL = r'https?://(?:(?P<kind>www|m)\.)?4tube\.com/(?:videos|embed)/(?P<id>\d+)(?:/(?P<display_id>[^/?#&]+))?'
     _URL_TEMPLATE = 'https://www.4tube.com/videos/%s/video'
-    _TKN_HOST = 'token.4tube.com'
     _TESTS = [{
         'url': 'http://www.4tube.com/videos/209733/hot-babe-holly-michaels-gets-her-ass-stuffed-by-black',
         'md5': '6516c8ac63b03de06bc8eac14362db4f',
@@ -148,7 +149,6 @@ class FourTubeIE(FourTubeBaseIE):
 class FuxIE(FourTubeBaseIE):
     _VALID_URL = r'https?://(?:(?P<kind>www|m)\.)?fux\.com/(?:video|embed)/(?P<id>\d+)(?:/(?P<display_id>[^/?#&]+))?'
     _URL_TEMPLATE = 'https://www.fux.com/video/%s/video'
-    _TKN_HOST = 'token.fux.com'
     _TESTS = [{
         'url': 'https://www.fux.com/video/195359/awesome-fucking-kitchen-ends-cum-swallow',
         'info_dict': {
@@ -280,7 +280,6 @@ class PornTubeIE(FourTubeBaseIE):
 class PornerBrosIE(FourTubeBaseIE):
     _VALID_URL = r'https?://(?:(?P<kind>www|m)\.)?pornerbros\.com/(?:videos/(?P<display_id>[^/]+)_|embed/)(?P<id>\d+)'
     _URL_TEMPLATE = 'https://www.pornerbros.com/videos/video_%s'
-    _TKN_HOST = 'token.pornerbros.com'
     _TESTS = [{
         'url': 'https://www.pornerbros.com/videos/skinny-brunette-takes-big-cock-down-her-anal-hole_181369',
         'md5': '6516c8ac63b03de06bc8eac14362db4f',

youtube_dl/extractor/fox.py

@@ -6,12 +6,10 @@ import uuid
 
 from .adobepass import AdobePassIE
 from ..compat import (
-    compat_HTTPError,
     compat_str,
     compat_urllib_parse_unquote,
 )
 from ..utils import (
-    ExtractorError,
     int_or_none,
     parse_age_limit,
     parse_duration,
@@ -50,7 +48,6 @@ class FOXIE(AdobePassIE):
         'url': 'https://www.fox.com/watch/30056b295fb57f7452aeeb4920bc3024/',
         'only_matching': True,
     }]
-    _GEO_BYPASS = False
     _HOME_PAGE_URL = 'https://www.fox.com/'
     _API_KEY = 'abdcbed02c124d393b39e818a4312055'
     _access_token = None
@@ -61,22 +58,9 @@ class FOXIE(AdobePassIE):
         }
         if self._access_token:
             headers['Authorization'] = 'Bearer ' + self._access_token
-        try:
-            return self._download_json(
-                'https://api2.fox.com/v2.0/' + path,
-                video_id, data=data, headers=headers)
-        except ExtractorError as e:
-            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
-                entitlement_issues = self._parse_json(
-                    e.cause.read().decode(), video_id)['entitlementIssues']
-                for e in entitlement_issues:
-                    if e.get('errorCode') == 1005:
-                        raise ExtractorError(
-                            'This video is only available via cable service provider '
-                            'subscription. You may want to use --cookies.', expected=True)
-                messages = ', '.join([e['message'] for e in entitlement_issues])
-                raise ExtractorError(messages, expected=True)
-            raise
+        return self._download_json(
+            'https://api2.fox.com/v2.0/' + path,
+            video_id, data=data, headers=headers)
 
     def _real_initialize(self):
         if not self._access_token:
@@ -97,15 +81,7 @@ class FOXIE(AdobePassIE):
 
         title = video['name']
         release_url = video['url']
-        try:
-            m3u8_url = self._download_json(release_url, video_id)['playURL']
-        except ExtractorError as e:
-            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
-                error = self._parse_json(e.cause.read().decode(), video_id)
-                if error.get('exception') == 'GeoLocationBlocked':
-                    self.raise_geo_restricted(countries=['US'])
-                raise ExtractorError(error['description'], expected=True)
-            raise
+        m3u8_url = self._download_json(release_url, video_id)['playURL']
         formats = self._extract_m3u8_formats(
             m3u8_url, video_id, 'mp4',
             entry_protocol='m3u8_native', m3u8_id='hls')

youtube_dl/extractor/francetv.py

@@ -371,13 +371,12 @@ class FranceTVInfoIE(FranceTVBaseInfoExtractor):
                 self.url_result(dailymotion_url, DailymotionIE.ie_key())
                 for dailymotion_url in dailymotion_urls])
 
-        video_id = self._search_regex(
-            (r'player\.load[^;]+src:\s*["\']([^"\']+)',
-             r'id-video=([^@]+@[^"]+)',
+        video_id, catalogue = self._search_regex(
+            (r'id-video=([^@]+@[^"]+)',
              r'<a[^>]+href="(?:https?:)?//videos\.francetv\.fr/video/([^@]+@[^"]+)"'),
-            webpage, 'video id')
+            webpage, 'video id').split('@')
 
-        return self._make_url_result(video_id)
+        return self._make_url_result(video_id, catalogue)
 
 
 class FranceTVInfoSportIE(FranceTVBaseInfoExtractor):

youtube_dl/extractor/frontendmasters.py

@@ -94,8 +94,8 @@ class FrontendMastersPageBaseIE(FrontendMastersBaseIE):
         chapter_number = None
         index = lesson.get('index')
         element_index = lesson.get('elementIndex')
-        if (isinstance(index, int) and isinstance(element_index, int)
-                and index < element_index):
+        if (isinstance(index, int) and isinstance(element_index, int) and
+                index < element_index):
             chapter_number = element_index - index
         chapter = (chapters[chapter_number - 1]
                    if chapter_number - 1 < len(chapters) else None)

youtube_dl/extractor/gaia.py

@@ -4,17 +4,12 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from ..compat import (
-    compat_str,
-    compat_urllib_parse_unquote,
-)
+from ..compat import compat_str
 from ..utils import (
-    ExtractorError,
     int_or_none,
     str_or_none,
     strip_or_none,
     try_get,
-    urlencode_postdata,
 )
 
 
@@ -51,29 +46,6 @@ class GaiaIE(InfoExtractor):
             'skip_download': True,
         },
     }]
-    _NETRC_MACHINE = 'gaia'
-    _jwt = None
-
-    def _real_initialize(self):
-        auth = self._get_cookies('https://www.gaia.com/').get('auth')
-        if auth:
-            auth = self._parse_json(
-                compat_urllib_parse_unquote(auth.value),
-                None, fatal=False)
-        if not auth:
-            username, password = self._get_login_info()
-            if username is None:
-                return
-            auth = self._download_json(
-                'https://auth.gaia.com/v1/login',
-                None, data=urlencode_postdata({
-                    'username': username,
-                    'password': password
-                }))
-            if auth.get('success') is False:
-                raise ExtractorError(', '.join(auth['messages']), expected=True)
-        if auth:
-            self._jwt = auth.get('jwt')
 
     def _real_extract(self, url):
         display_id, vtype = re.search(self._VALID_URL, url).groups()
@@ -87,12 +59,8 @@ class GaiaIE(InfoExtractor):
         media_id = compat_str(vdata['nid'])
         title = node['title']
 
-        headers = None
-        if self._jwt:
-            headers = {'Authorization': 'Bearer ' + self._jwt}
         media = self._download_json(
-            'https://brooklyn.gaia.com/media/' + media_id,
-            media_id, headers=headers)
+            'https://brooklyn.gaia.com/media/' + media_id, media_id)
         formats = self._extract_m3u8_formats(
             media['mediaUrls']['bcHLS'], media_id, 'mp4')
         self._sort_formats(formats)

youtube_dl/extractor/gdcvault.py

@@ -3,24 +3,22 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from .kaltura import KalturaIE
 from ..utils import (
     HEADRequest,
     sanitized_Request,
-    smuggle_url,
     urlencode_postdata,
 )
 
 
 class GDCVaultIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?gdcvault\.com/play/(?P<id>\d+)(?:/(?P<name>[\w-]+))?'
+    _VALID_URL = r'https?://(?:www\.)?gdcvault\.com/play/(?P<id>\d+)/(?P<name>(\w|-)+)?'
     _NETRC_MACHINE = 'gdcvault'
     _TESTS = [
         {
             'url': 'http://www.gdcvault.com/play/1019721/Doki-Doki-Universe-Sweet-Simple',
             'md5': '7ce8388f544c88b7ac11c7ab1b593704',
             'info_dict': {
-                'id': '201311826596_AWNY',
+                'id': '1019721',
                 'display_id': 'Doki-Doki-Universe-Sweet-Simple',
                 'ext': 'mp4',
                 'title': 'Doki-Doki Universe: Sweet, Simple and Genuine (GDC Next 10)'
@@ -29,7 +27,7 @@ class GDCVaultIE(InfoExtractor):
         {
             'url': 'http://www.gdcvault.com/play/1015683/Embracing-the-Dark-Art-of',
             'info_dict': {
-                'id': '201203272_1330951438328RSXR',
+                'id': '1015683',
                 'display_id': 'Embracing-the-Dark-Art-of',
                 'ext': 'flv',
                 'title': 'Embracing the Dark Art of Mathematical Modeling in AI'
@@ -58,7 +56,7 @@ class GDCVaultIE(InfoExtractor):
             'url': 'http://gdcvault.com/play/1023460/Tenacious-Design-and-The-Interface',
             'md5': 'a8efb6c31ed06ca8739294960b2dbabd',
             'info_dict': {
-                'id': '840376_BQRC',
+                'id': '1023460',
                 'ext': 'mp4',
                 'display_id': 'Tenacious-Design-and-The-Interface',
                 'title': 'Tenacious Design and The Interface of \'Destiny\'',
@@ -68,38 +66,26 @@ class GDCVaultIE(InfoExtractor):
             # Multiple audios
             'url': 'http://www.gdcvault.com/play/1014631/Classic-Game-Postmortem-PAC',
             'info_dict': {
-                'id': '12396_1299111843500GMPX',
-                'ext': 'mp4',
+                'id': '1014631',
+                'ext': 'flv',
                 'title': 'How to Create a Good Game - From My Experience of Designing Pac-Man',
             },
-            # 'params': {
-            #     'skip_download': True,  # Requires rtmpdump
-            #     'format': 'jp',  # The japanese audio
-            # }
+            'params': {
+                'skip_download': True,  # Requires rtmpdump
+                'format': 'jp',  # The japanese audio
+            }
         },
         {
             # gdc-player.html
             'url': 'http://www.gdcvault.com/play/1435/An-American-engine-in-Tokyo',
             'info_dict': {
-                'id': '9350_1238021887562UHXB',
+                'id': '1435',
                 'display_id': 'An-American-engine-in-Tokyo',
-                'ext': 'mp4',
+                'ext': 'flv',
                 'title': 'An American Engine in Tokyo:/nThe collaboration of Epic Games and Square Enix/nFor THE LAST REMINANT',
             },
-        },
-        {
-            # Kaltura Embed
-            'url': 'https://www.gdcvault.com/play/1026180/Mastering-the-Apex-of-Scaling',
-            'info_dict': {
-                'id': '0_h1fg8j3p',
-                'ext': 'mp4',
-                'title': 'Mastering the Apex of Scaling Game Servers (Presented by Multiplay)',
-                'timestamp': 1554401811,
-                'upload_date': '20190404',
-                'uploader_id': 'joe@blazestreaming.com',
-            },
             'params': {
-                'format': 'mp4-408',
+                'skip_download': True,  # Requires rtmpdump
             },
         },
     ]
@@ -128,8 +114,10 @@ class GDCVaultIE(InfoExtractor):
         return start_page
 
     def _real_extract(self, url):
-        video_id, name = re.match(self._VALID_URL, url).groups()
-        display_id = name or video_id
+        mobj = re.match(self._VALID_URL, url)
+
+        video_id = mobj.group('id')
+        display_id = mobj.group('name') or video_id
 
         webpage_url = 'http://www.gdcvault.com/play/' + video_id
         start_page = self._download_webpage(webpage_url, display_id)
@@ -139,12 +127,12 @@ class GDCVaultIE(InfoExtractor):
             start_page, 'url', default=None)
         if direct_url:
             title = self._html_search_regex(
-                r'<td><strong>Session Name:?</strong></td>\s*<td>(.*?)</td>',
+                r'<td><strong>Session Name</strong></td>\s*<td>(.*?)</td>',
                 start_page, 'title')
             video_url = 'http://www.gdcvault.com' + direct_url
             # resolve the url so that we can detect the correct extension
-            video_url = self._request_webpage(
-                HEADRequest(video_url), video_id).geturl()
+            head = self._request_webpage(HEADRequest(video_url), video_id)
+            video_url = head.geturl()
 
             return {
                 'id': video_id,
@@ -153,36 +141,34 @@ class GDCVaultIE(InfoExtractor):
                 'title': title,
             }
 
-        embed_url = KalturaIE._extract_url(start_page)
-        if embed_url:
-            embed_url = smuggle_url(embed_url, {'source_url': url})
-            ie_key = 'Kaltura'
-        else:
-            PLAYER_REGEX = r'<iframe src="(?P<xml_root>.+?)/(?:gdc-)?player.*?\.html.*?".*?</iframe>'
+        PLAYER_REGEX = r'<iframe src="(?P<xml_root>.+?)/(?:gdc-)?player.*?\.html.*?".*?</iframe>'
 
-            xml_root = self._html_search_regex(
-                PLAYER_REGEX, start_page, 'xml root', default=None)
-            if xml_root is None:
-                # Probably need to authenticate
-                login_res = self._login(webpage_url, display_id)
-                if login_res is None:
-                    self.report_warning('Could not login.')
-                else:
-                    start_page = login_res
-                    # Grab the url from the authenticated page
-                    xml_root = self._html_search_regex(
-                        PLAYER_REGEX, start_page, 'xml root')
+        xml_root = self._html_search_regex(
+            PLAYER_REGEX, start_page, 'xml root', default=None)
+        if xml_root is None:
+            # Probably need to authenticate
+            login_res = self._login(webpage_url, display_id)
+            if login_res is None:
+                self.report_warning('Could not login.')
+            else:
+                start_page = login_res
+                # Grab the url from the authenticated page
+                xml_root = self._html_search_regex(
+                    PLAYER_REGEX, start_page, 'xml root')
 
+        xml_name = self._html_search_regex(
+            r'<iframe src=".*?\?xml=(.+?\.xml).*?".*?</iframe>',
+            start_page, 'xml filename', default=None)
+        if xml_name is None:
+            # Fallback to the older format
             xml_name = self._html_search_regex(
-                r'<iframe src=".*?\?xml(?:=|URL=xml/)(.+?\.xml).*?".*?</iframe>',
+                r'<iframe src=".*?\?xmlURL=xml/(?P<xml_file>.+?\.xml).*?".*?</iframe>',
                 start_page, 'xml filename')
-            embed_url = '%s/xml/%s' % (xml_root, xml_name)
-            ie_key = 'DigitallySpeaking'
 
         return {
             '_type': 'url_transparent',
             'id': video_id,
             'display_id': display_id,
-            'url': embed_url,
-            'ie_key': ie_key,
+            'url': '%s/xml/%s' % (xml_root, xml_name),
+            'ie_key': 'DigitallySpeaking',
         }

youtube_dl/extractor/generic.py

@@ -89,10 +89,7 @@ from .piksel import PikselIE
 from .videa import VideaIE
 from .twentymin import TwentyMinutenIE
 from .ustream import UstreamIE
-from .openload import (
-    OpenloadIE,
-    VerystreamIE,
-)
+from .openload import OpenloadIE
 from .videopress import VideoPressIE
 from .rutube import RutubeIE
 from .limelight import LimelightBaseIE
@@ -2549,11 +2546,11 @@ class GenericIE(InfoExtractor):
             return self.url_result(mobj.group('url'))
 
         # Look for Ooyala videos
-        mobj = (re.search(r'player\.ooyala\.com/[^"?]+[?#][^"]*?(?:embedCode|ec)=(?P<ec>[^"&]+)', webpage)
-                or re.search(r'OO\.Player\.create\([\'"].*?[\'"],\s*[\'"](?P<ec>.{32})[\'"]', webpage)
-                or re.search(r'OO\.Player\.create\.apply\(\s*OO\.Player\s*,\s*op\(\s*\[\s*[\'"][^\'"]*[\'"]\s*,\s*[\'"](?P<ec>.{32})[\'"]', webpage)
-                or re.search(r'SBN\.VideoLinkset\.ooyala\([\'"](?P<ec>.{32})[\'"]\)', webpage)
-                or re.search(r'data-ooyala-video-id\s*=\s*[\'"](?P<ec>.{32})[\'"]', webpage))
+        mobj = (re.search(r'player\.ooyala\.com/[^"?]+[?#][^"]*?(?:embedCode|ec)=(?P<ec>[^"&]+)', webpage) or
+                re.search(r'OO\.Player\.create\([\'"].*?[\'"],\s*[\'"](?P<ec>.{32})[\'"]', webpage) or
+                re.search(r'OO\.Player\.create\.apply\(\s*OO\.Player\s*,\s*op\(\s*\[\s*[\'"][^\'"]*[\'"]\s*,\s*[\'"](?P<ec>.{32})[\'"]', webpage) or
+                re.search(r'SBN\.VideoLinkset\.ooyala\([\'"](?P<ec>.{32})[\'"]\)', webpage) or
+                re.search(r'data-ooyala-video-id\s*=\s*[\'"](?P<ec>.{32})[\'"]', webpage))
         if mobj is not None:
             embed_token = self._search_regex(
                 r'embedToken[\'"]?\s*:\s*[\'"]([^\'"]+)',
@@ -2583,6 +2580,19 @@ class GenericIE(InfoExtractor):
         if mobj is not None:
             return self.url_result(mobj.group(1), 'Mpora')
 
+        # Look for embedded NovaMov-based player
+        mobj = re.search(
+            r'''(?x)<(?:pagespeed_)?iframe[^>]+?src=(["\'])
+                    (?P<url>http://(?:(?:embed|www)\.)?
+                        (?:novamov\.com|
+                           nowvideo\.(?:ch|sx|eu|at|ag|co)|
+                           videoweed\.(?:es|com)|
+                           movshare\.(?:net|sx|ag)|
+                           divxstage\.(?:eu|net|ch|co|at|ag))
+                        /embed\.php.+?)\1''', webpage)
+        if mobj is not None:
+            return self.url_result(mobj.group('url'))
+
         # Look for embedded Facebook player
         facebook_urls = FacebookIE._extract_urls(webpage)
         if facebook_urls:
@@ -3007,12 +3017,6 @@ class GenericIE(InfoExtractor):
             return self.playlist_from_matches(
                 openload_urls, video_id, video_title, ie=OpenloadIE.ie_key())
 
-        # Look for Verystream embeds
-        verystream_urls = VerystreamIE._extract_urls(webpage)
-        if verystream_urls:
-            return self.playlist_from_matches(
-                verystream_urls, video_id, video_title, ie=VerystreamIE.ie_key())
-
         # Look for VideoPress embeds
         videopress_urls = VideoPressIE._extract_urls(webpage)
         if videopress_urls:
@@ -3208,8 +3212,8 @@ class GenericIE(InfoExtractor):
                 else:
                     formats.append({
                         'url': src,
-                        'ext': (mimetype2ext(src_type)
-                                or ext if ext in KNOWN_EXTENSIONS else 'mp4'),
+                        'ext': (mimetype2ext(src_type) or
+                                ext if ext in KNOWN_EXTENSIONS else 'mp4'),
                     })
             if formats:
                 self._sort_formats(formats)

youtube_dl/extractor/gfycat.py

@@ -11,7 +11,7 @@ from ..utils import (
 
 
 class GfycatIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?gfycat\.com/(?:ifr/|gifs/detail/)?(?P<id>[^-/?#]+)'
+    _VALID_URL = r'https?://(?:www\.)?gfycat\.com/(?:ifr/|gifs/detail/)?(?P<id>[^/?#]+)'
     _TESTS = [{
         'url': 'http://gfycat.com/DeadlyDecisiveGermanpinscher',
         'info_dict': {
@@ -47,9 +47,6 @@ class GfycatIE(InfoExtractor):
     }, {
         'url': 'https://gfycat.com/gifs/detail/UnconsciousLankyIvorygull',
         'only_matching': True
-    }, {
-        'url': 'https://gfycat.com/acceptablehappygoluckyharborporpoise-baseball',
-        'only_matching': True
     }]
 
     def _real_extract(self, url):

youtube_dl/extractor/hbo.py

@@ -4,12 +4,12 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
+from ..compat import compat_str
 from ..utils import (
     xpath_text,
     xpath_element,
     int_or_none,
     parse_duration,
-    urljoin,
 )
 
 
@@ -53,13 +53,10 @@ class HBOBaseIE(InfoExtractor):
         },
     }
 
-    def _extract_info(self, url, display_id):
-        video_data = self._download_xml(url, display_id)
-        video_id = xpath_text(video_data, 'id', fatal=True)
-        episode_title = title = xpath_text(video_data, 'title', fatal=True)
-        series = xpath_text(video_data, 'program')
-        if series:
-            title = '%s - %s' % (series, title)
+    def _extract_from_id(self, video_id):
+        video_data = self._download_xml(
+            'http://render.lv3.hbo.com/data/content/global/videos/data/%s.xml' % video_id, video_id)
+        title = xpath_text(video_data, 'title', 'title', True)
 
         formats = []
         for source in xpath_element(video_data, 'videos', 'sources', True):
@@ -131,45 +128,68 @@ class HBOBaseIE(InfoExtractor):
                     'width': width,
                 })
 
-        subtitles = None
-        caption_url = xpath_text(video_data, 'captionUrl')
-        if caption_url:
-            subtitles = {
-                'en': [{
-                    'url': caption_url,
-                    'ext': 'ttml'
-                }],
-            }
-
         return {
             'id': video_id,
             'title': title,
             'duration': parse_duration(xpath_text(video_data, 'duration/tv14')),
-            'series': series,
-            'episode': episode_title,
             'formats': formats,
             'thumbnails': thumbnails,
-            'subtitles': subtitles,
         }
 
 
 class HBOIE(HBOBaseIE):
     IE_NAME = 'hbo'
-    _VALID_URL = r'https?://(?:www\.)?hbo\.com/(?:video|embed)(?:/[^/]+)*/(?P<id>[^/?#]+)'
+    _VALID_URL = r'https?://(?:www\.)?hbo\.com/video/video\.html\?.*vid=(?P<id>[0-9]+)'
     _TEST = {
-        'url': 'https://www.hbo.com/video/game-of-thrones/seasons/season-8/videos/trailer',
-        'md5': '8126210656f433c452a21367f9ad85b3',
+        'url': 'http://www.hbo.com/video/video.html?autoplay=true&g=u&vid=1437839',
+        'md5': '2c6a6bc1222c7e91cb3334dad1746e5a',
         'info_dict': {
-            'id': '22113301',
+            'id': '1437839',
             'ext': 'mp4',
-            'title': 'Game of Thrones - Trailer',
-        },
-        'expected_warnings': ['Unknown MIME type application/mp4 in DASH manifest'],
+            'title': 'Ep. 64 Clip: Encryption',
+            'thumbnail': r're:https?://.*\.jpg$',
+            'duration': 1072,
+        }
     }
 
     def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-        location_path = self._parse_json(self._html_search_regex(
-            r'data-state="({.+?})"', webpage, 'state'), display_id)['video']['locationUrl']
-        return self._extract_info(urljoin(url, location_path), display_id)
+        video_id = self._match_id(url)
+        return self._extract_from_id(video_id)
+
+
+class HBOEpisodeIE(HBOBaseIE):
+    IE_NAME = 'hbo:episode'
+    _VALID_URL = r'https?://(?:www\.)?hbo\.com/(?P<path>(?!video)(?:(?:[^/]+/)+video|watch-free-episodes)/(?P<id>[0-9a-z-]+))(?:\.html)?'
+
+    _TESTS = [{
+        'url': 'http://www.hbo.com/girls/episodes/5/52-i-love-you-baby/video/ep-52-inside-the-episode.html?autoplay=true',
+        'md5': '61ead79b9c0dfa8d3d4b07ef4ac556fb',
+        'info_dict': {
+            'id': '1439518',
+            'display_id': 'ep-52-inside-the-episode',
+            'ext': 'mp4',
+            'title': 'Ep. 52: Inside the Episode',
+            'thumbnail': r're:https?://.*\.jpg$',
+            'duration': 240,
+        },
+    }, {
+        'url': 'http://www.hbo.com/game-of-thrones/about/video/season-5-invitation-to-the-set.html?autoplay=true',
+        'only_matching': True,
+    }, {
+        'url': 'http://www.hbo.com/watch-free-episodes/last-week-tonight-with-john-oliver',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        path, display_id = re.match(self._VALID_URL, url).groups()
+
+        content = self._download_json(
+            'http://www.hbo.com/api/content/' + path, display_id)['content']
+
+        video_id = compat_str((content.get('parsed', {}).get(
+            'common:FullBleedVideo', {}) or content['selectedEpisode'])['videoId'])
+
+        info_dict = self._extract_from_id(video_id)
+        info_dict['display_id'] = display_id
+
+        return info_dict

youtube_dl/extractor/heise.py

@@ -155,8 +155,8 @@ class HeiseIE(InfoExtractor):
             'id': video_id,
             'title': title,
             'description': description,
-            'thumbnail': (xpath_text(doc, './/{http://rss.jwpcdn.com/}image')
-                          or self._og_search_thumbnail(webpage)),
+            'thumbnail': (xpath_text(doc, './/{http://rss.jwpcdn.com/}image') or
+                          self._og_search_thumbnail(webpage)),
             'timestamp': parse_iso8601(
                 self._html_search_meta('date', webpage)),
             'formats': formats,

youtube_dl/extractor/hitbox.py

@@ -58,8 +58,8 @@ class HitboxIE(InfoExtractor):
         title = video_meta.get('media_status')
         alt_title = video_meta.get('media_title')
         description = clean_html(
-            video_meta.get('media_description')
-            or video_meta.get('media_description_md'))
+            video_meta.get('media_description') or
+            video_meta.get('media_description_md'))
         duration = float_or_none(video_meta.get('media_duration'))
         uploader = video_meta.get('media_user_name')
         views = int_or_none(video_meta.get('media_views'))

youtube_dl/extractor/hitrecord.py

@@ -47,8 +47,8 @@ class HitRecordIE(InfoExtractor):
             tags = [
                 t['text']
                 for t in tags_list
-                if isinstance(t, dict) and t.get('text')
-                and isinstance(t['text'], compat_str)]
+                if isinstance(t, dict) and t.get('text') and
+                isinstance(t['text'], compat_str)]
 
         return {
             'id': video_id,

youtube_dl/extractor/hketv.py

@@ -77,13 +77,13 @@ class HKETVIE(InfoExtractor):
 
         title = (
             self._html_search_meta(
-                ('ed_title', 'search.ed_title'), webpage, default=None)
-            or self._search_regex(
+                ('ed_title', 'search.ed_title'), webpage, default=None) or
+            self._search_regex(
                 r'data-favorite_title_(?:eng|chi)=(["\'])(?P<id>(?:(?!\1).)+)\1',
-                webpage, 'title', default=None, group='url')
-            or self._html_search_regex(
-                r'<h1>([^<]+)</h1>', webpage, 'title', default=None)
-            or self._og_search_title(webpage)
+                webpage, 'title', default=None, group='url') or
+            self._html_search_regex(
+                r'<h1>([^<]+)</h1>', webpage, 'title', default=None) or
+            self._og_search_title(webpage)
         )
 
         file_id = self._search_regex(

youtube_dl/extractor/hotstar.py

@@ -4,59 +4,40 @@ from __future__ import unicode_literals
 import hashlib
 import hmac
 import time
-import uuid
 
 from .common import InfoExtractor
-from ..compat import (
-    compat_HTTPError,
-    compat_str,
-)
+from ..compat import compat_HTTPError
 from ..utils import (
     determine_ext,
     ExtractorError,
     int_or_none,
-    str_or_none,
     try_get,
-    url_or_none,
 )
 
 
 class HotStarBaseIE(InfoExtractor):
     _AKAMAI_ENCRYPTION_KEY = b'\x05\xfc\x1a\x01\xca\xc9\x4b\xc4\x12\xfc\x53\x12\x07\x75\xf9\xee'
 
-    def _call_api_impl(self, path, video_id, query):
+    def _call_api(self, path, video_id, query_name='contentId'):
         st = int(time.time())
         exp = st + 6000
         auth = 'st=%d~exp=%d~acl=/*' % (st, exp)
         auth += '~hmac=' + hmac.new(self._AKAMAI_ENCRYPTION_KEY, auth.encode(), hashlib.sha256).hexdigest()
         response = self._download_json(
-            'https://api.hotstar.com/' + path, video_id, headers={
+            'https://api.hotstar.com/' + path,
+            video_id, headers={
                 'hotstarauth': auth,
                 'x-country-code': 'IN',
                 'x-platform-code': 'JIO',
-            }, query=query)
+            }, query={
+                query_name: video_id,
+                'tas': 10000,
+            })
         if response['statusCode'] != 'OK':
             raise ExtractorError(
                 response['body']['message'], expected=True)
         return response['body']['results']
 
-    def _call_api(self, path, video_id, query_name='contentId'):
-        return self._call_api_impl(path, video_id, {
-            query_name: video_id,
-            'tas': 10000,
-        })
-
-    def _call_api_v2(self, path, video_id):
-        return self._call_api_impl(
-            '%s/in/contents/%s' % (path, video_id), video_id, {
-                'desiredConfig': 'encryption:plain;ladder:phone,tv;package:hls,dash',
-                'client': 'mweb',
-                'clientVersion': '6.18.0',
-                'deviceId': compat_str(uuid.uuid4()),
-                'osName': 'Windows',
-                'osVersion': '10',
-            })
-
 
 class HotStarIE(HotStarBaseIE):
     IE_NAME = 'hotstar'
@@ -87,10 +68,6 @@ class HotStarIE(HotStarBaseIE):
     }, {
         'url': 'http://www.hotstar.com/1000000515',
         'only_matching': True,
-    }, {
-        # only available via api v2
-        'url': 'https://www.hotstar.com/tv/ek-bhram-sarvagun-sampanna/s-2116/janhvi-targets-suman/1000234847',
-        'only_matching': True,
     }]
     _GEO_BYPASS = False
 
@@ -118,40 +95,26 @@ class HotStarIE(HotStarBaseIE):
             raise ExtractorError('This video is DRM protected.', expected=True)
 
         formats = []
-        geo_restricted = False
-        playback_sets = self._call_api_v2('h/v2/play', video_id)['playBackSets']
-        for playback_set in playback_sets:
-            if not isinstance(playback_set, dict):
-                continue
-            format_url = url_or_none(playback_set.get('playbackUrl'))
-            if not format_url:
-                continue
-            tags = str_or_none(playback_set.get('tagsCombination')) or ''
-            if tags and 'encryption:plain' not in tags:
-                continue
-            ext = determine_ext(format_url)
+        format_data = self._call_api('h/v1/play', video_id)['item']
+        format_url = format_data['playbackUrl']
+        ext = determine_ext(format_url)
+        if ext == 'm3u8':
             try:
-                if 'package:hls' in tags or ext == 'm3u8':
-                    formats.extend(self._extract_m3u8_formats(
-                        format_url, video_id, 'mp4', m3u8_id='hls'))
-                elif 'package:dash' in tags or ext == 'mpd':
-                    formats.extend(self._extract_mpd_formats(
-                        format_url, video_id, mpd_id='dash'))
-                elif ext == 'f4m':
-                    # produce broken files
-                    pass
-                else:
-                    formats.append({
-                        'url': format_url,
-                        'width': int_or_none(playback_set.get('width')),
-                        'height': int_or_none(playback_set.get('height')),
-                    })
+                formats.extend(self._extract_m3u8_formats(
+                    format_url, video_id, 'mp4', m3u8_id='hls'))
             except ExtractorError as e:
                 if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
-                    geo_restricted = True
-                continue
-        if not formats and geo_restricted:
-            self.raise_geo_restricted(countries=['IN'])
+                    self.raise_geo_restricted(countries=['IN'])
+                raise
+        elif ext == 'f4m':
+            # produce broken files
+            pass
+        else:
+            formats.append({
+                'url': format_url,
+                'width': int_or_none(format_data.get('width')),
+                'height': int_or_none(format_data.get('height')),
+            })
         self._sort_formats(formats)
 
         return {

youtube_dl/extractor/hrti.py

@@ -60,8 +60,8 @@ class HRTiBaseIE(InfoExtractor):
             language=self._APP_LANGUAGE,
             application_id=self._APP_PUBLICATION_ID)
 
-        self._login_url = (modules['user']['resources']['login']['uri']
-                           + '/format/json').format(session_id=self._session_id)
+        self._login_url = (modules['user']['resources']['login']['uri'] +
+                           '/format/json').format(session_id=self._session_id)
 
         self._logout_url = modules['user']['resources']['logout']['uri']
 

youtube_dl/extractor/hypem.py

@@ -1,11 +1,18 @@
 from __future__ import unicode_literals
 
+import json
+import time
+
 from .common import InfoExtractor
-from ..utils import int_or_none
+from ..compat import compat_urllib_parse_urlencode
+from ..utils import (
+    ExtractorError,
+    sanitized_Request,
+)
 
 
 class HypemIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?hypem\.com/track/(?P<id>[0-9a-z]{5})'
+    _VALID_URL = r'https?://(?:www\.)?hypem\.com/track/(?P<id>[^/]+)/'
     _TEST = {
         'url': 'http://hypem.com/track/1v6ga/BODYWORK+-+TAME',
         'md5': 'b9cc91b5af8995e9f0c1cee04c575828',
@@ -14,36 +21,41 @@ class HypemIE(InfoExtractor):
             'ext': 'mp3',
             'title': 'Tame',
             'uploader': 'BODYWORK',
-            'timestamp': 1371810457,
-            'upload_date': '20130621',
         }
     }
 
     def _real_extract(self, url):
         track_id = self._match_id(url)
 
-        response = self._download_webpage(url, track_id)
+        data = {'ax': 1, 'ts': time.time()}
+        request = sanitized_Request(url + '?' + compat_urllib_parse_urlencode(data))
+        response, urlh = self._download_webpage_handle(
+            request, track_id, 'Downloading webpage with the url')
 
-        track = self._parse_json(self._html_search_regex(
-            r'(?s)<script\s+type="application/json"\s+id="displayList-data">(.+?)</script>',
-            response, 'tracks'), track_id)['tracks'][0]
+        html_tracks = self._html_search_regex(
+            r'(?ms)<script type="application/json" id="displayList-data">(.+?)</script>',
+            response, 'tracks')
+        try:
+            track_list = json.loads(html_tracks)
+            track = track_list['tracks'][0]
+        except ValueError:
+            raise ExtractorError('Hypemachine contained invalid JSON.')
 
+        key = track['key']
         track_id = track['id']
         title = track['song']
 
-        final_url = self._download_json(
-            'http://hypem.com/serve/source/%s/%s' % (track_id, track['key']),
-            track_id, 'Downloading metadata', headers={
-                'Content-Type': 'application/json'
-            })['url']
+        request = sanitized_Request(
+            'http://hypem.com/serve/source/%s/%s' % (track_id, key),
+            '', {'Content-Type': 'application/json'})
+        song_data = self._download_json(request, track_id, 'Downloading metadata')
+        final_url = song_data['url']
+        artist = track.get('artist')
 
         return {
             'id': track_id,
             'url': final_url,
             'ext': 'mp3',
             'title': title,
-            'uploader': track.get('artist'),
-            'duration': int_or_none(track.get('time')),
-            'timestamp': int_or_none(track.get('ts')),
-            'track': title,
+            'uploader': artist,
         }

youtube_dl/extractor/ina.py

@@ -1,83 +1,36 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import re
+
 from .common import InfoExtractor
-from ..utils import (
-    determine_ext,
-    int_or_none,
-    strip_or_none,
-    xpath_attr,
-    xpath_text,
-)
 
 
 class InaIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?ina\.fr/(?:video|audio)/(?P<id>[A-Z0-9_]+)'
-    _TESTS = [{
+    _VALID_URL = r'https?://(?:www\.)?ina\.fr/video/(?P<id>I?[A-Z0-9]+)'
+    _TEST = {
         'url': 'http://www.ina.fr/video/I12055569/francois-hollande-je-crois-que-c-est-clair-video.html',
         'md5': 'a667021bf2b41f8dc6049479d9bb38a3',
         'info_dict': {
             'id': 'I12055569',
             'ext': 'mp4',
             'title': 'François Hollande "Je crois que c\'est clair"',
-            'description': 'md5:3f09eb072a06cb286b8f7e4f77109663',
         }
-    }, {
-        'url': 'https://www.ina.fr/video/S806544_001/don-d-organes-des-avancees-mais-d-importants-besoins-video.html',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.ina.fr/audio/P16173408',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.ina.fr/video/P16173408-video.html',
-        'only_matching': True,
-    }]
+    }
 
     def _real_extract(self, url):
-        video_id = self._match_id(url)
-        info_doc = self._download_xml(
-            'http://player.ina.fr/notices/%s.mrss' % video_id, video_id)
-        item = info_doc.find('channel/item')
-        title = xpath_text(item, 'title', fatal=True)
-        media_ns_xpath = lambda x: self._xpath_ns(x, 'http://search.yahoo.com/mrss/')
-        content = item.find(media_ns_xpath('content'))
+        mobj = re.match(self._VALID_URL, url)
 
-        get_furl = lambda x: xpath_attr(content, media_ns_xpath(x), 'url')
-        formats = []
-        for q, w, h in (('bq', 400, 300), ('mq', 512, 384), ('hq', 768, 576)):
-            q_url = get_furl(q)
-            if not q_url:
-                continue
-            formats.append({
-                'format_id': q,
-                'url': q_url,
-                'width': w,
-                'height': h,
-            })
-        if not formats:
-            furl = get_furl('player') or content.attrib['url']
-            ext = determine_ext(furl)
-            formats = [{
-                'url': furl,
-                'vcodec': 'none' if ext == 'mp3' else None,
-                'ext': ext,
-            }]
+        video_id = mobj.group('id')
+        mrss_url = 'http://player.ina.fr/notices/%s.mrss' % video_id
+        info_doc = self._download_xml(mrss_url, video_id)
 
-        thumbnails = []
-        for thumbnail in content.findall(media_ns_xpath('thumbnail')):
-            thumbnail_url = thumbnail.get('url')
-            if not thumbnail_url:
-                continue
-            thumbnails.append({
-                'url': thumbnail_url,
-                'height': int_or_none(thumbnail.get('height')),
-                'width': int_or_none(thumbnail.get('width')),
-            })
+        self.report_extraction(video_id)
+
+        video_url = info_doc.find('.//{http://search.yahoo.com/mrss/}player').attrib['url']
 
         return {
             'id': video_id,
-            'formats': formats,
-            'title': title,
-            'description': strip_or_none(xpath_text(item, 'description')),
-            'thumbnails': thumbnails,
+            'url': video_url,
+            'title': info_doc.find('.//title').text,
         }

youtube_dl/extractor/infoq.py

@@ -122,9 +122,9 @@ class InfoQIE(BokeCCBaseIE):
             formats = self._extract_bokecc_formats(webpage, video_id)
         else:
             formats = (
-                self._extract_rtmp_video(webpage)
-                + self._extract_http_video(webpage)
-                + self._extract_http_audio(webpage, video_id))
+                self._extract_rtmp_video(webpage) +
+                self._extract_http_video(webpage) +
+                self._extract_http_audio(webpage, video_id))
 
         self._sort_formats(formats)
 

youtube_dl/extractor/iqiyi.py

@@ -383,9 +383,9 @@ class IqiyiIE(InfoExtractor):
             self._sleep(5, video_id)
 
         self._sort_formats(formats)
-        title = (get_element_by_id('widget-videotitle', webpage)
-                 or clean_html(get_element_by_attribute('class', 'mod-play-tit', webpage))
-                 or self._html_search_regex(r'<span[^>]+data-videochanged-title="word"[^>]*>([^<]+)</span>', webpage, 'title'))
+        title = (get_element_by_id('widget-videotitle', webpage) or
+                 clean_html(get_element_by_attribute('class', 'mod-play-tit', webpage)) or
+                 self._html_search_regex(r'<span[^>]+data-videochanged-title="word"[^>]*>([^<]+)</span>', webpage, 'title'))
 
         return {
             'id': video_id,

youtube_dl/extractor/itv.py

@@ -77,10 +77,10 @@ class ITVIE(InfoExtractor):
             return etree.SubElement(element, _add_ns(name))
 
         production_id = (
-            params.get('data-video-autoplay-id')
-            or '%s#001' % (
-                params.get('data-video-episode-id')
-                or video_id.replace('a', '/')))
+            params.get('data-video-autoplay-id') or
+            '%s#001' % (
+                params.get('data-video-episode-id') or
+                video_id.replace('a', '/')))
 
         req_env = etree.Element(_add_ns('soapenv:Envelope'))
         _add_sub_element(req_env, 'soapenv:Header')

youtube_dl/extractor/jwplatform.py

@@ -7,7 +7,7 @@ from .common import InfoExtractor
 
 
 class JWPlatformIE(InfoExtractor):
-    _VALID_URL = r'(?:https?://(?:content\.jwplatform|cdn\.jwplayer)\.com/(?:(?:feed|player|thumb|preview|video)s|jw6|v2/media)/|jwplatform:)(?P<id>[a-zA-Z0-9]{8})'
+    _VALID_URL = r'(?:https?://(?:content\.jwplatform|cdn\.jwplayer)\.com/(?:(?:feed|player|thumb|preview|video|manifest)s|jw6|v2/media)/|jwplatform:)(?P<id>[a-zA-Z0-9]{8})'
     _TESTS = [{
         'url': 'http://content.jwplatform.com/players/nPripu9l-ALJ3XQCI.js',
         'md5': 'fa8899fa601eb7c83a64e9d568bdf325',

youtube_dl/extractor/kaltura.py

@@ -118,8 +118,8 @@ class KalturaIE(InfoExtractor):
                         (?P<q2>['"])_?(?P<partner_id>(?:(?!(?P=q2)).)+)(?P=q2),.*?
                         (?P<q3>['"])entry_?[Ii]d(?P=q3)\s*:\s*
                         (?P<q4>['"])(?P<id>(?:(?!(?P=q4)).)+)(?P=q4)(?:,|\s*\})
-                """, webpage)
-            or re.search(
+                """, webpage) or
+            re.search(
                 r'''(?xs)
                     (?P<q1>["'])
                         (?:https?:)?//cdnapi(?:sec)?\.kaltura\.com(?::\d+)?/(?:(?!(?P=q1)).)*\b(?:p|partner_id)/(?P<partner_id>\d+)(?:(?!(?P=q1)).)*
@@ -132,8 +132,8 @@ class KalturaIE(InfoExtractor):
                         \[\s*(?P<q2_1>["'])entry_?[Ii]d(?P=q2_1)\s*\]\s*=\s*
                     )
                     (?P<q3>["'])(?P<id>(?:(?!(?P=q3)).)+)(?P=q3)
-                ''', webpage)
-            or re.search(
+                ''', webpage) or
+            re.search(
                 r'''(?xs)
                     <(?:iframe[^>]+src|meta[^>]+\bcontent)=(?P<q1>["'])
                       (?:https?:)?//(?:(?:www|cdnapi(?:sec)?)\.)?kaltura\.com/(?:(?!(?P=q1)).)*\b(?:p|partner_id)/(?P<partner_id>\d+)
@@ -145,8 +145,6 @@ class KalturaIE(InfoExtractor):
         )
         if mobj:
             embed_info = mobj.groupdict()
-            for k, v in embed_info.items():
-                embed_info[k] = v.strip()
             url = 'kaltura:%(partner_id)s:%(id)s' % embed_info
             escaped_pid = re.escape(embed_info['partner_id'])
             service_url = re.search(

youtube_dl/extractor/karrierevideos.py

@@ -47,8 +47,8 @@ class KarriereVideosIE(InfoExtractor):
 
         webpage = self._download_webpage(url, video_id)
 
-        title = (self._html_search_meta('title', webpage, default=None)
-                 or self._search_regex(r'<h1 class="title">([^<]+)</h1>', webpage, 'video title'))
+        title = (self._html_search_meta('title', webpage, default=None) or
+                 self._search_regex(r'<h1 class="title">([^<]+)</h1>', webpage, 'video title'))
 
         video_id = self._search_regex(
             r'/config/video/(.+?)\.xml', webpage, 'video id')

youtube_dl/extractor/linkedin.py

@@ -9,13 +9,11 @@ from ..utils import (
     float_or_none,
     int_or_none,
     urlencode_postdata,
-    urljoin,
 )
 
 
 class LinkedInLearningBaseIE(InfoExtractor):
     _NETRC_MACHINE = 'linkedin'
-    _LOGIN_URL = 'https://www.linkedin.com/uas/login?trk=learning'
 
     def _call_api(self, course_slug, fields, video_slug=None, resolution=None):
         query = {
@@ -52,10 +50,11 @@ class LinkedInLearningBaseIE(InfoExtractor):
             return
 
         login_page = self._download_webpage(
-            self._LOGIN_URL, None, 'Downloading login page')
-        action_url = urljoin(self._LOGIN_URL, self._search_regex(
+            'https://www.linkedin.com/uas/login?trk=learning',
+            None, 'Downloading login page')
+        action_url = self._search_regex(
             r'<form[^>]+action=(["\'])(?P<url>.+?)\1', login_page, 'post url',
-            default='https://www.linkedin.com/uas/login-submit', group='url'))
+            default='https://www.linkedin.com/uas/login-submit', group='url')
         data = self._hidden_inputs(login_page)
         data.update({
             'session_key': email,

youtube_dl/extractor/liveleak.py

@@ -82,10 +82,6 @@ class LiveLeakIE(InfoExtractor):
     }, {
         'url': 'https://www.liveleak.com/view?t=HvHi_1523016227',
         'only_matching': True,
-    }, {
-        # No original video
-        'url': 'https://www.liveleak.com/view?t=C26ZZ_1558612804',
-        'only_matching': True,
     }]
 
     @staticmethod
@@ -138,13 +134,11 @@ class LiveLeakIE(InfoExtractor):
                 orig_url = re.sub(r'\.mp4\.[^.]+', '', a_format['url'])
                 if a_format['url'] != orig_url:
                     format_id = a_format.get('format_id')
-                    format_id = 'original' + ('-' + format_id if format_id else '')
-                    if self._is_valid_url(orig_url, video_id, format_id):
-                        formats.append({
-                            'format_id': format_id,
-                            'url': orig_url,
-                            'preference': 1,
-                        })
+                    formats.append({
+                        'format_id': 'original' + ('-' + format_id if format_id else ''),
+                        'url': orig_url,
+                        'preference': 1,
+                    })
             self._sort_formats(formats)
             info_dict['formats'] = formats
 

youtube_dl/extractor/mediasite.py

@@ -13,8 +13,6 @@ from ..utils import (
     ExtractorError,
     float_or_none,
     mimetype2ext,
-    str_or_none,
-    try_get,
     unescapeHTML,
     unsmuggle_url,
     url_or_none,
@@ -22,11 +20,8 @@ from ..utils import (
 )
 
 
-_ID_RE = r'(?:[0-9a-f]{32,34}|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12,14})'
-
-
 class MediasiteIE(InfoExtractor):
-    _VALID_URL = r'(?xi)https?://[^/]+/Mediasite/(?:Play|Showcase/(?:default|livebroadcast)/Presentation)/(?P<id>%s)(?P<query>\?[^#]+|)' % _ID_RE
+    _VALID_URL = r'(?xi)https?://[^/]+/Mediasite/(?:Play|Showcase/(?:default|livebroadcast)/Presentation)/(?P<id>[0-9a-f]{32,34})(?P<query>\?[^#]+|)'
     _TESTS = [
         {
             'url': 'https://hitsmediaweb.h-its.org/mediasite/Play/2db6c271681e4f199af3c60d1f82869b1d',
@@ -98,11 +93,6 @@ class MediasiteIE(InfoExtractor):
             'url': 'https://mediasite.ntnu.no/Mediasite/Showcase/default/Presentation/7d8b913259334b688986e970fae6fcb31d',
             'only_matching': True,
         },
-        {
-            # dashed id
-            'url': 'https://hitsmediaweb.h-its.org/mediasite/Play/2db6c271-681e-4f19-9af3-c60d1f82869b1d',
-            'only_matching': True,
-        }
     ]
 
     # look in Mediasite.Core.js (Mediasite.ContentStreamType[*])
@@ -119,7 +109,7 @@ class MediasiteIE(InfoExtractor):
         return [
             unescapeHTML(mobj.group('url'))
             for mobj in re.finditer(
-                r'(?xi)<iframe\b[^>]+\bsrc=(["\'])(?P<url>(?:(?:https?:)?//[^/]+)?/Mediasite/Play/%s(?:\?.*?)?)\1' % _ID_RE,
+                r'(?xi)<iframe\b[^>]+\bsrc=(["\'])(?P<url>(?:(?:https?:)?//[^/]+)?/Mediasite/Play/[0-9a-f]{32,34}(?:\?.*?)?)\1',
                 webpage)]
 
     def _real_extract(self, url):
@@ -231,136 +221,3 @@ class MediasiteIE(InfoExtractor):
             'formats': formats,
             'thumbnails': thumbnails,
         }
-
-
-class MediasiteCatalogIE(InfoExtractor):
-    _VALID_URL = r'''(?xi)
-                        (?P<url>https?://[^/]+/Mediasite)
-                        /Catalog/Full/
-                        (?P<catalog_id>{0})
-                        (?:
-                            /(?P<current_folder_id>{0})
-                            /(?P<root_dynamic_folder_id>{0})
-                        )?
-                    '''.format(_ID_RE)
-    _TESTS = [{
-        'url': 'http://events7.mediasite.com/Mediasite/Catalog/Full/631f9e48530d454381549f955d08c75e21',
-        'info_dict': {
-            'id': '631f9e48530d454381549f955d08c75e21',
-            'title': 'WCET Summit: Adaptive Learning in Higher Ed: Improving Outcomes Dynamically',
-        },
-        'playlist_count': 6,
-        'expected_warnings': ['is not a supported codec'],
-    }, {
-        # with CurrentFolderId and RootDynamicFolderId
-        'url': 'https://medaudio.medicine.iu.edu/Mediasite/Catalog/Full/9518c4a6c5cf4993b21cbd53e828a92521/97a9db45f7ab47428c77cd2ed74bb98f14/9518c4a6c5cf4993b21cbd53e828a92521',
-        'info_dict': {
-            'id': '9518c4a6c5cf4993b21cbd53e828a92521',
-            'title': 'IUSM Family and Friends Sessions',
-        },
-        'playlist_count': 2,
-    }, {
-        'url': 'http://uipsyc.mediasite.com/mediasite/Catalog/Full/d5d79287c75243c58c50fef50174ec1b21',
-        'only_matching': True,
-    }, {
-        # no AntiForgeryToken
-        'url': 'https://live.libraries.psu.edu/Mediasite/Catalog/Full/8376d4b24dd1457ea3bfe4cf9163feda21',
-        'only_matching': True,
-    }, {
-        'url': 'https://medaudio.medicine.iu.edu/Mediasite/Catalog/Full/9518c4a6c5cf4993b21cbd53e828a92521/97a9db45f7ab47428c77cd2ed74bb98f14/9518c4a6c5cf4993b21cbd53e828a92521',
-        'only_matching': True,
-    }, {
-        # dashed id
-        'url': 'http://events7.mediasite.com/Mediasite/Catalog/Full/631f9e48-530d-4543-8154-9f955d08c75e',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        mediasite_url = mobj.group('url')
-        catalog_id = mobj.group('catalog_id')
-        current_folder_id = mobj.group('current_folder_id') or catalog_id
-        root_dynamic_folder_id = mobj.group('root_dynamic_folder_id')
-
-        webpage = self._download_webpage(url, catalog_id)
-
-        # AntiForgeryToken is optional (e.g. [1])
-        # 1. https://live.libraries.psu.edu/Mediasite/Catalog/Full/8376d4b24dd1457ea3bfe4cf9163feda21
-        anti_forgery_token = self._search_regex(
-            r'AntiForgeryToken\s*:\s*(["\'])(?P<value>(?:(?!\1).)+)\1',
-            webpage, 'anti forgery token', default=None, group='value')
-        if anti_forgery_token:
-            anti_forgery_header = self._search_regex(
-                r'AntiForgeryHeaderName\s*:\s*(["\'])(?P<value>(?:(?!\1).)+)\1',
-                webpage, 'anti forgery header name',
-                default='X-SOFO-AntiForgeryHeader', group='value')
-
-        data = {
-            'IsViewPage': True,
-            'IsNewFolder': True,
-            'AuthTicket': None,
-            'CatalogId': catalog_id,
-            'CurrentFolderId': current_folder_id,
-            'RootDynamicFolderId': root_dynamic_folder_id,
-            'ItemsPerPage': 1000,
-            'PageIndex': 0,
-            'PermissionMask': 'Execute',
-            'CatalogSearchType': 'SearchInFolder',
-            'SortBy': 'Date',
-            'SortDirection': 'Descending',
-            'StartDate': None,
-            'EndDate': None,
-            'StatusFilterList': None,
-            'PreviewKey': None,
-            'Tags': [],
-        }
-
-        headers = {
-            'Content-Type': 'application/json; charset=UTF-8',
-            'Referer': url,
-            'X-Requested-With': 'XMLHttpRequest',
-        }
-        if anti_forgery_token:
-            headers[anti_forgery_header] = anti_forgery_token
-
-        catalog = self._download_json(
-            '%s/Catalog/Data/GetPresentationsForFolder' % mediasite_url,
-            catalog_id, data=json.dumps(data).encode(), headers=headers)
-
-        entries = []
-        for video in catalog['PresentationDetailsList']:
-            if not isinstance(video, dict):
-                continue
-            video_id = str_or_none(video.get('Id'))
-            if not video_id:
-                continue
-            entries.append(self.url_result(
-                '%s/Play/%s' % (mediasite_url, video_id),
-                ie=MediasiteIE.ie_key(), video_id=video_id))
-
-        title = try_get(
-            catalog, lambda x: x['CurrentFolder']['Name'], compat_str)
-
-        return self.playlist_result(entries, catalog_id, title,)
-
-
-class MediasiteNamedCatalogIE(InfoExtractor):
-    _VALID_URL = r'(?xi)(?P<url>https?://[^/]+/Mediasite)/Catalog/catalogs/(?P<catalog_name>[^/?#&]+)'
-    _TESTS = [{
-        'url': 'https://msite.misis.ru/Mediasite/Catalog/catalogs/2016-industrial-management-skriabin-o-o',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        mediasite_url = mobj.group('url')
-        catalog_name = mobj.group('catalog_name')
-
-        webpage = self._download_webpage(url, catalog_name)
-
-        catalog_id = self._search_regex(
-            r'CatalogId\s*:\s*["\'](%s)' % _ID_RE, webpage, 'catalog id')
-
-        return self.url_result(
-            '%s/Catalog/Full/%s' % (mediasite_url, catalog_id),
-            ie=MediasiteCatalogIE.ie_key(), video_id=catalog_id)

youtube_dl/extractor/mgtv.py

@@ -1,32 +1,22 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import base64
-import time
-import uuid
-
 from .common import InfoExtractor
-from ..compat import (
-    compat_HTTPError,
-    compat_str,
-)
-from ..utils import (
-    ExtractorError,
-    int_or_none,
-)
+from ..compat import compat_str
+from ..utils import int_or_none
 
 
 class MGTVIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?mgtv\.com/(v|b)/(?:[^/]+/)*(?P<id>\d+)\.html'
     IE_DESC = '芒果TV'
-    _GEO_COUNTRIES = ['CN']
 
     _TESTS = [{
         'url': 'http://www.mgtv.com/v/1/290525/f/3116640.html',
+        'md5': 'b1ffc0fc163152acf6beaa81832c9ee7',
         'info_dict': {
             'id': '3116640',
             'ext': 'mp4',
-            'title': '我是歌手 第四季',
+            'title': '我是歌手第四季双年巅峰会：韩红李玟“双王”领军对抗',
             'description': '我是歌手第四季双年巅峰会',
             'duration': 7461,
             'thumbnail': r're:^https?://.*\.jpg$',
@@ -38,30 +28,16 @@ class MGTVIE(InfoExtractor):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        try:
-            api_data = self._download_json(
-                'https://pcweb.api.mgtv.com/player/video', video_id, query={
-                    'tk2': base64.urlsafe_b64encode(b'did=%s|pno=1030|ver=0.3.0301|clit=%d' % (compat_str(uuid.uuid4()).encode(), time.time()))[::-1],
-                    'video_id': video_id,
-                }, headers=self.geo_verification_headers())['data']
-        except ExtractorError as e:
-            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
-                error = self._parse_json(e.cause.read().decode(), None)
-                if error.get('code') == 40005:
-                    self.raise_geo_restricted(countries=self._GEO_COUNTRIES)
-                raise ExtractorError(error['msg'], expected=True)
-            raise
+        api_data = self._download_json(
+            'http://pcweb.api.mgtv.com/player/video', video_id,
+            query={'video_id': video_id},
+            headers=self.geo_verification_headers())['data']
         info = api_data['info']
         title = info['title'].strip()
-        stream_data = self._download_json(
-            'https://pcweb.api.mgtv.com/player/getSource', video_id, query={
-                'pm2': api_data['atc']['pm2'],
-                'video_id': video_id,
-            }, headers=self.geo_verification_headers())['data']
-        stream_domain = stream_data['stream_domain'][0]
+        stream_domain = api_data['stream_domain'][0]
 
         formats = []
-        for idx, stream in enumerate(stream_data['stream']):
+        for idx, stream in enumerate(api_data['stream']):
             stream_path = stream.get('url')
             if not stream_path:
                 continue
@@ -71,7 +47,7 @@ class MGTVIE(InfoExtractor):
             format_url = format_data.get('info')
             if not format_url:
                 continue
-            tbr = int_or_none(stream.get('filebitrate') or self._search_regex(
+            tbr = int_or_none(self._search_regex(
                 r'_(\d+)_mp4/', format_url, 'tbr', default=None))
             formats.append({
                 'format_id': compat_str(tbr or idx),

youtube_dl/extractor/moevideo.py

@@ -1,12 +1,15 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import json
 import re
 
 from .common import InfoExtractor
 from ..utils import (
-    clean_html,
+    ExtractorError,
     int_or_none,
+    sanitized_Request,
+    urlencode_postdata,
 )
 
 
@@ -14,8 +17,8 @@ class MoeVideoIE(InfoExtractor):
     IE_DESC = 'LetitBit video services: moevideo.net, playreplay.net and videochart.net'
     _VALID_URL = r'''(?x)
         https?://(?P<host>(?:www\.)?
-        (?:(?:moevideo|playreplay|videochart)\.net|thesame\.tv))/
-        (?:video|framevideo|embed)/(?P<id>[0-9a-z]+\.[0-9A-Za-z]+)'''
+        (?:(?:moevideo|playreplay|videochart)\.net))/
+        (?:video|framevideo)/(?P<id>[0-9]+\.[0-9A-Za-z]+)'''
     _API_URL = 'http://api.letitbit.net/'
     _API_KEY = 'tVL0gjqo5'
     _TESTS = [
@@ -54,26 +57,58 @@ class MoeVideoIE(InfoExtractor):
     ]
 
     def _real_extract(self, url):
-        host, video_id = re.match(self._VALID_URL, url).groups()
+        mobj = re.match(self._VALID_URL, url)
+        video_id = mobj.group('id')
 
         webpage = self._download_webpage(
-            'http://%s/video/%s' % (host, video_id),
+            'http://%s/video/%s' % (mobj.group('host'), video_id),
             video_id, 'Downloading webpage')
 
         title = self._og_search_title(webpage)
+        thumbnail = self._og_search_thumbnail(webpage)
+        description = self._og_search_description(webpage)
 
-        embed_webpage = self._download_webpage(
-            'http://%s/embed/%s' % (host, video_id),
-            video_id, 'Downloading embed webpage')
-        video = self._parse_json(self._search_regex(
-            r'mvplayer\("#player"\s*,\s*({.+})',
-            embed_webpage, 'mvplayer'), video_id)['video']
+        r = [
+            self._API_KEY,
+            [
+                'preview/flv_link',
+                {
+                    'uid': video_id,
+                },
+            ],
+        ]
+        r_json = json.dumps(r)
+        post = urlencode_postdata({'r': r_json})
+        req = sanitized_Request(self._API_URL, post)
+        req.add_header('Content-type', 'application/x-www-form-urlencoded')
+
+        response = self._download_json(req, video_id)
+        if response['status'] != 'OK':
+            raise ExtractorError(
+                '%s returned error: %s' % (self.IE_NAME, response['data']),
+                expected=True
+            )
+        item = response['data'][0]
+        video_url = item['link']
+        duration = int_or_none(item['length'])
+        width = int_or_none(item['width'])
+        height = int_or_none(item['height'])
+        filesize = int_or_none(item['convert_size'])
+
+        formats = [{
+            'format_id': 'sd',
+            'http_headers': {'Range': 'bytes=0-'},  # Required to download
+            'url': video_url,
+            'width': width,
+            'height': height,
+            'filesize': filesize,
+        }]
 
         return {
             'id': video_id,
             'title': title,
-            'thumbnail': video.get('poster') or self._og_search_thumbnail(webpage),
-            'description': clean_html(self._og_search_description(webpage)),
-            'duration': int_or_none(self._og_search_property('video:duration', webpage)),
-            'url': video['ourUrl'],
+            'thumbnail': thumbnail,
+            'description': description,
+            'duration': duration,
+            'formats': formats,
         }

youtube_dl/extractor/motherless.py

@@ -80,8 +80,8 @@ class MotherlessIE(InfoExtractor):
         video_url = (self._html_search_regex(
             (r'setup\(\{\s*["\']file["\']\s*:\s*(["\'])(?P<url>(?:(?!\1).)+)\1',
              r'fileurl\s*=\s*(["\'])(?P<url>(?:(?!\1).)+)\1'),
-            webpage, 'video URL', default=None, group='url')
-            or 'http://cdn4.videos.motherlessmedia.com/videos/%s.mp4?fs=opencloud' % video_id)
+            webpage, 'video URL', default=None, group='url') or
+            'http://cdn4.videos.motherlessmedia.com/videos/%s.mp4?fs=opencloud' % video_id)
         age_limit = self._rta_search(webpage)
         view_count = str_to_int(self._html_search_regex(
             r'<strong>Views</strong>\s+([^<]+)<',