Compare commits
1 Commits
2016.06.19
...
2016.06.04
| Author | SHA1 | Date | |
|---|---|---|---|
|
4ea8d2ee76 |
6
.github/ISSUE_TEMPLATE.md
vendored
6
.github/ISSUE_TEMPLATE.md
vendored
@@ -6,8 +6,8 @@
|
||||
|
||||
---
|
||||
|
||||
### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2016.06.19.1*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
|
||||
- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2016.06.19.1**
|
||||
### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2016.06.04*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
|
||||
- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2016.06.04**
|
||||
|
||||
### Before submitting an *issue* make sure you have:
|
||||
- [ ] At least skimmed through [README](https://github.com/rg3/youtube-dl/blob/master/README.md) and **most notably** [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
|
||||
@@ -35,7 +35,7 @@ $ youtube-dl -v <your command line>
|
||||
[debug] User config: []
|
||||
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
|
||||
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
|
||||
[debug] youtube-dl version 2016.06.19.1
|
||||
[debug] youtube-dl version 2016.06.04
|
||||
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
|
||||
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
|
||||
[debug] Proxy map: {}
|
||||
|
||||
2
AUTHORS
2
AUTHORS
@@ -173,5 +173,3 @@ Kevin Deldycke
|
||||
inondle
|
||||
Tomáš Čech
|
||||
Déstin Reed
|
||||
Roman Tsiupa
|
||||
Artur Krysiak
|
||||
|
||||
@@ -142,9 +142,9 @@ After you have ensured this site is distributing it's content legally, you can f
|
||||
```
|
||||
5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
|
||||
6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc.
|
||||
7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L74-L252). Add tests and code for as many as you want.
|
||||
8. Keep in mind that the only mandatory fields in info dict for successful extraction process are `id`, `title` and either `url` or `formats`, i.e. these are the critical data the extraction does not make any sense without. This means that [any field](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L148-L252) apart from aforementioned mandatory ones should be treated **as optional** and extraction should be **tolerate** to situations when sources for these fields can potentially be unavailable (even if they always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields. For example, if you have some intermediate dict `meta` that is a source of metadata and it has a key `summary` that you want to extract and put into resulting info dict as `description`, you should be ready that this key may be missing from the `meta` dict, i.e. you should extract it as `meta.get('summary')` and not `meta['summary']`. Similarly, you should pass `fatal=False` when extracting data from a webpage with `_search_regex/_html_search_regex`.
|
||||
9. Check the code with [flake8](https://pypi.python.org/pypi/flake8). Also make sure your code works under all [Python](http://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
|
||||
7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/58525c94d547be1c8167d16c298bdd75506db328/youtube_dl/extractor/common.py#L68-L226). Add tests and code for as many as you want.
|
||||
8. Keep in mind that the only mandatory fields in info dict for successful extraction process are `id`, `title` and either `url` or `formats`, i.e. these are the critical data the extraction does not make any sense without. This means that [any field](https://github.com/rg3/youtube-dl/blob/58525c94d547be1c8167d16c298bdd75506db328/youtube_dl/extractor/common.py#L138-L226) apart from aforementioned mandatory ones should be treated **as optional** and extraction should be **tolerate** to situations when sources for these fields can potentially be unavailable (even if they always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields. For example, if you have some intermediate dict `meta` that is a source of metadata and it has a key `summary` that you want to extract and put into resulting info dict as `description`, you should be ready that this key may be missing from the `meta` dict, i.e. you should extract it as `meta.get('summary')` and not `meta['summary']`. Similarly, you should pass `fatal=False` when extracting data from a webpage with `_search_regex/_html_search_regex`.
|
||||
9. Check the code with [flake8](https://pypi.python.org/pypi/flake8).
|
||||
10. When the tests pass, [add](http://git-scm.com/docs/git-add) the new files and [commit](http://git-scm.com/docs/git-commit) them and [push](http://git-scm.com/docs/git-push) the result, like this:
|
||||
|
||||
$ git add youtube_dl/extractor/extractors.py
|
||||
|
||||
39
README.md
39
README.md
@@ -17,7 +17,7 @@ youtube-dl - download videos from youtube.com or other video platforms
|
||||
|
||||
To install it right away for all UNIX users (Linux, OS X, etc.), type:
|
||||
|
||||
sudo curl -L https://yt-dl.org/latest/youtube-dl -o /usr/local/bin/youtube-dl
|
||||
sudo curl https://yt-dl.org/latest/youtube-dl -o /usr/local/bin/youtube-dl
|
||||
sudo chmod a+rx /usr/local/bin/youtube-dl
|
||||
|
||||
If you do not have curl, you can alternatively use a recent wget:
|
||||
@@ -27,24 +27,18 @@ If you do not have curl, you can alternatively use a recent wget:
|
||||
|
||||
Windows users can [download an .exe file](https://yt-dl.org/latest/youtube-dl.exe) and place it in any location on their [PATH](http://en.wikipedia.org/wiki/PATH_%28variable%29) except for `%SYSTEMROOT%\System32` (e.g. **do not** put in `C:\Windows\System32`).
|
||||
|
||||
You can also use pip:
|
||||
|
||||
sudo pip install --upgrade youtube-dl
|
||||
|
||||
This command will update youtube-dl if you have already installed it. See the [pypi page](https://pypi.python.org/pypi/youtube_dl) for more information.
|
||||
|
||||
OS X users can install youtube-dl with [Homebrew](http://brew.sh/):
|
||||
OS X users can install **youtube-dl** with [Homebrew](http://brew.sh/).
|
||||
|
||||
brew install youtube-dl
|
||||
|
||||
Or with [MacPorts](https://www.macports.org/):
|
||||
You can also use pip:
|
||||
|
||||
sudo port install youtube-dl
|
||||
sudo pip install youtube-dl
|
||||
|
||||
Alternatively, refer to the [developer instructions](#developer-instructions) for how to check out and work with the git repository. For further options, including PGP signatures, see the [youtube-dl Download Page](https://rg3.github.io/youtube-dl/download.html).
|
||||
|
||||
# DESCRIPTION
|
||||
**youtube-dl** is a command-line program to download videos from
|
||||
**youtube-dl** is a small command-line program to download videos from
|
||||
YouTube.com and a few more sites. It requires the Python interpreter, version
|
||||
2.6, 2.7, or 3.2+, and it is not platform specific. It should work on
|
||||
your Unix box, on Windows or on Mac OS X. It is released to the public domain,
|
||||
@@ -255,7 +249,7 @@ which means you can modify it, redistribute it or use it however you like.
|
||||
--write-info-json Write video metadata to a .info.json file
|
||||
--write-annotations Write video annotations to a
|
||||
.annotations.xml file
|
||||
--load-info-json FILE JSON file containing the video information
|
||||
--load-info FILE JSON file containing the video information
|
||||
(created with the "--write-info-json"
|
||||
option)
|
||||
--cookies FILE File to read cookies from and dump cookie
|
||||
@@ -511,9 +505,6 @@ The basic usage is not to set any template arguments when downloading a single f
|
||||
- `autonumber`: Five-digit number that will be increased with each download, starting at zero
|
||||
- `playlist`: Name or id of the playlist that contains the video
|
||||
- `playlist_index`: Index of the video in the playlist padded with leading zeros according to the total length of the playlist
|
||||
- `playlist_id`: Playlist identifier
|
||||
- `playlist_title`: Playlist title
|
||||
|
||||
|
||||
Available for the video that belongs to some logical chapter or section:
|
||||
- `chapter`: Name or title of the chapter the video belongs to
|
||||
@@ -553,10 +544,6 @@ The current default template is `%(title)s-%(id)s.%(ext)s`.
|
||||
|
||||
In some cases, you don't want special characters such as 中, spaces, or &, such as when transferring the downloaded filename to a Windows system or the filename through an 8bit-unsafe channel. In these cases, add the `--restrict-filenames` flag to get a shorter title:
|
||||
|
||||
#### Output template and Windows batch files
|
||||
|
||||
If you are using output template inside a Windows batch file then you must escape plain percent characters (`%`) by doubling, so that `-o "%(title)s-%(id)s.%(ext)s"` should become `-o "%%(title)s-%%(id)s.%%(ext)s"`. However you should not touch `%`'s that are not plain characters, e.g. environment variables for expansion should stay intact: `-o "C:\%HOMEPATH%\Desktop\%%(title)s.%%(ext)s"`.
|
||||
|
||||
#### Output template examples
|
||||
|
||||
Note on Windows you may need to use double quotes instead of single.
|
||||
@@ -855,12 +842,6 @@ It is *not* possible to detect whether a URL is supported or not. That's because
|
||||
|
||||
If you want to find out whether a given URL is supported, simply call youtube-dl with it. If you get no videos back, chances are the URL is either not referring to a video or unsupported. You can find out which by examining the output (if you run youtube-dl on the console) or catching an `UnsupportedError` exception if you run it from a Python program.
|
||||
|
||||
# Why do I need to go through that much red tape when filing bugs?
|
||||
|
||||
Before we had the issue template, despite our extensive [bug reporting instructions](#bugs), about 80% of the issue reports we got were useless, for instance because people used ancient versions hundreds of releases old, because of simple syntactic errors (not in youtube-dl but in general shell usage), because the problem was alrady reported multiple times before, because people did not actually read an error message, even if it said "please install ffmpeg", because people did not mention the URL they were trying to download and many more simple, easy-to-avoid problems, many of whom were totally unrelated to youtube-dl.
|
||||
|
||||
youtube-dl is an open-source project manned by too few volunteers, so we'd rather spend time fixing bugs where we are certain none of those simple problems apply, and where we can be reasonably confident to be able to reproduce the issue without asking the reporter repeatedly. As such, the output of `youtube-dl -v YOUR_URL_HERE` is really all that's required to file an issue. The issue template also guides you through some basic steps you can do, such as checking that your version of youtube-dl is current.
|
||||
|
||||
# DEVELOPER INSTRUCTIONS
|
||||
|
||||
Most users do not need to build youtube-dl and can [download the builds](http://rg3.github.io/youtube-dl/download.html) or get them from their distribution.
|
||||
@@ -935,9 +916,9 @@ After you have ensured this site is distributing it's content legally, you can f
|
||||
```
|
||||
5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
|
||||
6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc.
|
||||
7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L74-L252). Add tests and code for as many as you want.
|
||||
8. Keep in mind that the only mandatory fields in info dict for successful extraction process are `id`, `title` and either `url` or `formats`, i.e. these are the critical data the extraction does not make any sense without. This means that [any field](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L148-L252) apart from aforementioned mandatory ones should be treated **as optional** and extraction should be **tolerate** to situations when sources for these fields can potentially be unavailable (even if they always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields. For example, if you have some intermediate dict `meta` that is a source of metadata and it has a key `summary` that you want to extract and put into resulting info dict as `description`, you should be ready that this key may be missing from the `meta` dict, i.e. you should extract it as `meta.get('summary')` and not `meta['summary']`. Similarly, you should pass `fatal=False` when extracting data from a webpage with `_search_regex/_html_search_regex`.
|
||||
9. Check the code with [flake8](https://pypi.python.org/pypi/flake8). Also make sure your code works under all [Python](http://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
|
||||
7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/58525c94d547be1c8167d16c298bdd75506db328/youtube_dl/extractor/common.py#L68-L226). Add tests and code for as many as you want.
|
||||
8. Keep in mind that the only mandatory fields in info dict for successful extraction process are `id`, `title` and either `url` or `formats`, i.e. these are the critical data the extraction does not make any sense without. This means that [any field](https://github.com/rg3/youtube-dl/blob/58525c94d547be1c8167d16c298bdd75506db328/youtube_dl/extractor/common.py#L138-L226) apart from aforementioned mandatory ones should be treated **as optional** and extraction should be **tolerate** to situations when sources for these fields can potentially be unavailable (even if they always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields. For example, if you have some intermediate dict `meta` that is a source of metadata and it has a key `summary` that you want to extract and put into resulting info dict as `description`, you should be ready that this key may be missing from the `meta` dict, i.e. you should extract it as `meta.get('summary')` and not `meta['summary']`. Similarly, you should pass `fatal=False` when extracting data from a webpage with `_search_regex/_html_search_regex`.
|
||||
9. Check the code with [flake8](https://pypi.python.org/pypi/flake8).
|
||||
10. When the tests pass, [add](http://git-scm.com/docs/git-add) the new files and [commit](http://git-scm.com/docs/git-commit) them and [push](http://git-scm.com/docs/git-push) the result, like this:
|
||||
|
||||
$ git add youtube_dl/extractor/extractors.py
|
||||
@@ -964,7 +945,7 @@ with youtube_dl.YoutubeDL(ydl_opts) as ydl:
|
||||
ydl.download(['http://www.youtube.com/watch?v=BaW_jenozKc'])
|
||||
```
|
||||
|
||||
Most likely, you'll want to use various options. For a list of options available, have a look at [`youtube_dl/YoutubeDL.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/YoutubeDL.py#L128-L278). For a start, if you want to intercept youtube-dl's output, set a `logger` object.
|
||||
Most likely, you'll want to use various options. For a list of what can be done, have a look at [`youtube_dl/YoutubeDL.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/YoutubeDL.py#L121-L269). For a start, if you want to intercept youtube-dl's output, set a `logger` object.
|
||||
|
||||
Here's a more complete example of a program that outputs only errors (and a short message after the download is finished), and downloads/converts the video to an mp3 file:
|
||||
|
||||
|
||||
@@ -13,7 +13,6 @@ import os.path
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname((os.path.abspath(__file__)))))
|
||||
from youtube_dl.compat import (
|
||||
compat_input,
|
||||
compat_http_server,
|
||||
compat_str,
|
||||
compat_urlparse,
|
||||
@@ -31,6 +30,11 @@ try:
|
||||
except ImportError: # Python 2
|
||||
import SocketServer as compat_socketserver
|
||||
|
||||
try:
|
||||
compat_input = raw_input
|
||||
except NameError: # Python 3
|
||||
compat_input = input
|
||||
|
||||
|
||||
class BuildHTTPServer(compat_socketserver.ThreadingMixIn, compat_http_server.HTTPServer):
|
||||
allow_reuse_address = True
|
||||
|
||||
@@ -1,111 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import base64
|
||||
import json
|
||||
import mimetypes
|
||||
import netrc
|
||||
import optparse
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from youtube_dl.compat import (
|
||||
compat_basestring,
|
||||
compat_input,
|
||||
compat_getpass,
|
||||
compat_print,
|
||||
compat_urllib_request,
|
||||
)
|
||||
from youtube_dl.utils import (
|
||||
make_HTTPS_handler,
|
||||
sanitized_Request,
|
||||
)
|
||||
|
||||
|
||||
class GitHubReleaser(object):
|
||||
_API_URL = 'https://api.github.com/repos/rg3/youtube-dl/releases'
|
||||
_UPLOADS_URL = 'https://uploads.github.com/repos/rg3/youtube-dl/releases/%s/assets?name=%s'
|
||||
_NETRC_MACHINE = 'github.com'
|
||||
|
||||
def __init__(self, debuglevel=0):
|
||||
self._init_github_account()
|
||||
https_handler = make_HTTPS_handler({}, debuglevel=debuglevel)
|
||||
self._opener = compat_urllib_request.build_opener(https_handler)
|
||||
|
||||
def _init_github_account(self):
|
||||
try:
|
||||
info = netrc.netrc().authenticators(self._NETRC_MACHINE)
|
||||
if info is not None:
|
||||
self._username = info[0]
|
||||
self._password = info[2]
|
||||
compat_print('Using GitHub credentials found in .netrc...')
|
||||
return
|
||||
else:
|
||||
compat_print('No GitHub credentials found in .netrc')
|
||||
except (IOError, netrc.NetrcParseError):
|
||||
compat_print('Unable to parse .netrc')
|
||||
self._username = compat_input(
|
||||
'Type your GitHub username or email address and press [Return]: ')
|
||||
self._password = compat_getpass(
|
||||
'Type your GitHub password and press [Return]: ')
|
||||
|
||||
def _call(self, req):
|
||||
if isinstance(req, compat_basestring):
|
||||
req = sanitized_Request(req)
|
||||
# Authorizing manually since GitHub does not response with 401 with
|
||||
# WWW-Authenticate header set (see
|
||||
# https://developer.github.com/v3/#basic-authentication)
|
||||
b64 = base64.b64encode(
|
||||
('%s:%s' % (self._username, self._password)).encode('utf-8')).decode('ascii')
|
||||
req.add_header('Authorization', 'Basic %s' % b64)
|
||||
response = self._opener.open(req).read().decode('utf-8')
|
||||
return json.loads(response)
|
||||
|
||||
def list_releases(self):
|
||||
return self._call(self._API_URL)
|
||||
|
||||
def create_release(self, tag_name, name=None, body='', draft=False, prerelease=False):
|
||||
data = {
|
||||
'tag_name': tag_name,
|
||||
'target_commitish': 'master',
|
||||
'name': name,
|
||||
'body': body,
|
||||
'draft': draft,
|
||||
'prerelease': prerelease,
|
||||
}
|
||||
req = sanitized_Request(self._API_URL, json.dumps(data).encode('utf-8'))
|
||||
return self._call(req)
|
||||
|
||||
def create_asset(self, release_id, asset):
|
||||
asset_name = os.path.basename(asset)
|
||||
url = self._UPLOADS_URL % (release_id, asset_name)
|
||||
# Our files are small enough to be loaded directly into memory.
|
||||
data = open(asset, 'rb').read()
|
||||
req = sanitized_Request(url, data)
|
||||
mime_type, _ = mimetypes.guess_type(asset_name)
|
||||
req.add_header('Content-Type', mime_type or 'application/octet-stream')
|
||||
return self._call(req)
|
||||
|
||||
|
||||
def main():
|
||||
parser = optparse.OptionParser(usage='%prog VERSION BUILDPATH')
|
||||
options, args = parser.parse_args()
|
||||
if len(args) != 2:
|
||||
parser.error('Expected a version and a build directory')
|
||||
|
||||
version, build_path = args
|
||||
|
||||
releaser = GitHubReleaser()
|
||||
|
||||
new_release = releaser.create_release(version, name='youtube-dl %s' % version)
|
||||
release_id = new_release['id']
|
||||
|
||||
for asset in os.listdir(build_path):
|
||||
compat_print('Uploading %s...' % asset)
|
||||
releaser.create_asset(release_id, os.path.join(build_path, asset))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -15,7 +15,6 @@
|
||||
set -e
|
||||
|
||||
skip_tests=true
|
||||
gpg_sign_commits=""
|
||||
buildserver='localhost:8142'
|
||||
|
||||
while true
|
||||
@@ -25,10 +24,6 @@ case "$1" in
|
||||
skip_tests=false
|
||||
shift
|
||||
;;
|
||||
--gpg-sign-commits|-S)
|
||||
gpg_sign_commits="-S"
|
||||
shift
|
||||
;;
|
||||
--buildserver)
|
||||
buildserver="$2"
|
||||
shift 2
|
||||
@@ -74,7 +69,7 @@ sed -i "s/__version__ = '.*'/__version__ = '$version'/" youtube_dl/version.py
|
||||
/bin/echo -e "\n### Committing documentation, templates and youtube_dl/version.py..."
|
||||
make README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE.md supportedsites
|
||||
git add README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE.md docs/supportedsites.md youtube_dl/version.py
|
||||
git commit $gpg_sign_commits -m "release $version"
|
||||
git commit -m "release $version"
|
||||
|
||||
/bin/echo -e "\n### Now tagging, signing and pushing..."
|
||||
git tag -s -m "Release $version" "$version"
|
||||
@@ -100,16 +95,15 @@ RELEASE_FILES="youtube-dl youtube-dl.exe youtube-dl-$version.tar.gz"
|
||||
(cd build/$version/ && sha256sum $RELEASE_FILES > SHA2-256SUMS)
|
||||
(cd build/$version/ && sha512sum $RELEASE_FILES > SHA2-512SUMS)
|
||||
|
||||
/bin/echo -e "\n### Signing and uploading the new binaries to GitHub..."
|
||||
/bin/echo -e "\n### Signing and uploading the new binaries to yt-dl.org ..."
|
||||
for f in $RELEASE_FILES; do gpg --passphrase-repeat 5 --detach-sig "build/$version/$f"; done
|
||||
|
||||
ROOT=$(pwd)
|
||||
python devscripts/create-github-release.py $version "$ROOT/build/$version"
|
||||
|
||||
scp -r "build/$version" ytdl@yt-dl.org:html/tmp/
|
||||
ssh ytdl@yt-dl.org "mv html/tmp/$version html/downloads/"
|
||||
ssh ytdl@yt-dl.org "sh html/update_latest.sh $version"
|
||||
|
||||
/bin/echo -e "\n### Now switching to gh-pages..."
|
||||
git clone --branch gh-pages --single-branch . build/gh-pages
|
||||
ROOT=$(pwd)
|
||||
(
|
||||
set -e
|
||||
ORIGIN_URL=$(git config --get remote.origin.url)
|
||||
@@ -121,7 +115,7 @@ git clone --branch gh-pages --single-branch . build/gh-pages
|
||||
"$ROOT/devscripts/gh-pages/update-copyright.py"
|
||||
"$ROOT/devscripts/gh-pages/update-sites.py"
|
||||
git add *.html *.html.in update
|
||||
git commit $gpg_sign_commits -m "release $version"
|
||||
git commit -m "release $version"
|
||||
git push "$ROOT" gh-pages
|
||||
git push "$ORIGIN_URL" gh-pages
|
||||
)
|
||||
|
||||
@@ -28,7 +28,6 @@
|
||||
- **AdobeTVVideo**
|
||||
- **AdultSwim**
|
||||
- **aenetworks**: A+E Networks: A&E, Lifetime, History.com, FYI Network
|
||||
- **AfreecaTV**: afreecatv.com
|
||||
- **Aftonbladet**
|
||||
- **AirMozilla**
|
||||
- **AlJazeera**
|
||||
@@ -74,8 +73,6 @@
|
||||
- **bbc**: BBC
|
||||
- **bbc.co.uk**: BBC iPlayer
|
||||
- **bbc.co.uk:article**: BBC articles
|
||||
- **bbc.co.uk:iplayer:playlist**
|
||||
- **bbc.co.uk:playlist**
|
||||
- **BeatportPro**
|
||||
- **Beeg**
|
||||
- **BehindKink**
|
||||
@@ -106,8 +103,6 @@
|
||||
- **canalc2.tv**
|
||||
- **Canalplus**: canalplus.fr, piwiplus.fr and d8.tv
|
||||
- **Canvas**
|
||||
- **CarambaTV**
|
||||
- **CarambaTVPage**
|
||||
- **CBC**
|
||||
- **CBCPlayer**
|
||||
- **CBS**
|
||||
@@ -128,7 +123,6 @@
|
||||
- **cliphunter**
|
||||
- **ClipRs**
|
||||
- **Clipsyndicate**
|
||||
- **CloserToTruth**
|
||||
- **cloudtime**: CloudTime
|
||||
- **Cloudy**
|
||||
- **Clubic**
|
||||
@@ -259,7 +253,6 @@
|
||||
- **Globo**
|
||||
- **GloboArticle**
|
||||
- **GodTube**
|
||||
- **GodTV**
|
||||
- **GoldenMoustache**
|
||||
- **Golem**
|
||||
- **GoogleDrive**
|
||||
@@ -437,7 +430,6 @@
|
||||
- **nhl.com:videocenter**
|
||||
- **nhl.com:videocenter:category**: NHL videocenter category
|
||||
- **nick.com**
|
||||
- **nick.de**
|
||||
- **niconico**: ニコニコ動画
|
||||
- **NiconicoPlaylist**
|
||||
- **njoy**: N-JOY
|
||||
@@ -522,7 +514,6 @@
|
||||
- **qqmusic:singer**: QQ音乐 - 歌手
|
||||
- **qqmusic:toplist**: QQ音乐 - 排行榜
|
||||
- **R7**
|
||||
- **R7Article**
|
||||
- **radio.de**
|
||||
- **radiobremen**
|
||||
- **radiocanada**
|
||||
@@ -542,7 +533,6 @@
|
||||
- **revision3:embed**
|
||||
- **RICE**
|
||||
- **RingTV**
|
||||
- **RockstarGames**
|
||||
- **RottenTomatoes**
|
||||
- **Roxwel**
|
||||
- **RTBF**
|
||||
@@ -655,7 +645,6 @@
|
||||
- **Telegraaf**
|
||||
- **TeleMB**
|
||||
- **TeleTask**
|
||||
- **Telewebion**
|
||||
- **TF1**
|
||||
- **TheIntercept**
|
||||
- **ThePlatform**
|
||||
@@ -707,7 +696,6 @@
|
||||
- **TVPlay**: TV3Play and related services
|
||||
- **Tweakers**
|
||||
- **twitch:chapter**
|
||||
- **twitch:clips**
|
||||
- **twitch:past_broadcasts**
|
||||
- **twitch:profile**
|
||||
- **twitch:stream**
|
||||
@@ -750,7 +738,6 @@
|
||||
- **VideoPremium**
|
||||
- **VideoTt**: video.tt - Your True Tube (Currently broken)
|
||||
- **videoweed**: VideoWeed
|
||||
- **Vidio**
|
||||
- **vidme**
|
||||
- **vidme:user**
|
||||
- **vidme:user:likes**
|
||||
@@ -786,6 +773,7 @@
|
||||
- **VRT**
|
||||
- **vube**: Vube.com
|
||||
- **VuClip**
|
||||
- **vulture.com**
|
||||
- **Walla**
|
||||
- **washingtonpost**
|
||||
- **washingtonpost:article**
|
||||
@@ -793,8 +781,10 @@
|
||||
- **WatchIndianPorn**: Watch Indian Porn
|
||||
- **WDR**
|
||||
- **wdr:mobile**
|
||||
- **WDRMaus**: Sendung mit der Maus
|
||||
- **WebOfStories**
|
||||
- **WebOfStoriesPlaylist**
|
||||
- **Weibo**
|
||||
- **WeiqiTV**: WQTV
|
||||
- **wholecloud**: WholeCloud
|
||||
- **Wimp**
|
||||
@@ -802,11 +792,10 @@
|
||||
- **WNL**
|
||||
- **WorldStarHipHop**
|
||||
- **wrzuta.pl**
|
||||
- **wrzuta.pl:playlist**
|
||||
- **WSJ**: Wall Street Journal
|
||||
- **XBef**
|
||||
- **XboxClips**
|
||||
- **XFileShare**: XFileShare based sites: DaClips, FileHoot, GorillaVid, MovPod, PowerWatch, Rapidvideo.ws, TheVideoBee, Vidto, Streamin.To, XVIDSTAGE
|
||||
- **XFileShare**: XFileShare based sites: DaClips, FileHoot, GorillaVid, MovPod, PowerWatch, Rapidvideo.ws, TheVideoBee, Vidto, Streamin.To
|
||||
- **XHamster**
|
||||
- **XHamsterEmbed**
|
||||
- **xiami:album**: 虾米音乐 - 专辑
|
||||
@@ -831,7 +820,6 @@
|
||||
- **Ynet**
|
||||
- **YouJizz**
|
||||
- **youku**: 优酷
|
||||
- **youku:show**
|
||||
- **YouPorn**
|
||||
- **YourUpload**
|
||||
- **youtube**: YouTube.com
|
||||
|
||||
1
setup.py
1
setup.py
@@ -122,7 +122,6 @@ setup(
|
||||
"Programming Language :: Python :: 3.2",
|
||||
"Programming Language :: Python :: 3.3",
|
||||
"Programming Language :: Python :: 3.4",
|
||||
"Programming Language :: Python :: 3.5",
|
||||
],
|
||||
|
||||
cmdclass={'build_lazy_extractors': build_lazy_extractors},
|
||||
|
||||
@@ -249,8 +249,6 @@ class TestUtil(unittest.TestCase):
|
||||
self.assertEqual(unescapeHTML('/'), '/')
|
||||
self.assertEqual(unescapeHTML('é'), 'é')
|
||||
self.assertEqual(unescapeHTML('�'), '�')
|
||||
# HTML5 entities
|
||||
self.assertEqual(unescapeHTML('.''), '.\'')
|
||||
|
||||
def test_date_from_str(self):
|
||||
self.assertEqual(date_from_str('yesterday'), date_from_str('now-1day'))
|
||||
@@ -640,9 +638,6 @@ class TestUtil(unittest.TestCase):
|
||||
"1":{"src":"skipped", "type": "application/vnd.apple.mpegURL"}
|
||||
}''')
|
||||
|
||||
inp = '''{"foo":101}'''
|
||||
self.assertEqual(js_to_json(inp), '''{"foo":101}''')
|
||||
|
||||
def test_js_to_json_edgecases(self):
|
||||
on = js_to_json("{abc_def:'1\\'\\\\2\\\\\\'3\"4'}")
|
||||
self.assertEqual(json.loads(on), {"abc_def": "1'\\2\\'3\"4"})
|
||||
|
||||
@@ -1223,10 +1223,6 @@ class YoutubeDL(object):
|
||||
if 'title' not in info_dict:
|
||||
raise ExtractorError('Missing "title" field in extractor result')
|
||||
|
||||
if not isinstance(info_dict['id'], compat_str):
|
||||
self.report_warning('"id" field is not a string - forcing string conversion')
|
||||
info_dict['id'] = compat_str(info_dict['id'])
|
||||
|
||||
if 'playlist' not in info_dict:
|
||||
# It isn't part of a playlist
|
||||
info_dict['playlist'] = None
|
||||
|
||||
@@ -18,6 +18,7 @@ from .options import (
|
||||
from .compat import (
|
||||
compat_expanduser,
|
||||
compat_getpass,
|
||||
compat_print,
|
||||
compat_shlex_split,
|
||||
workaround_optparse_bug9161,
|
||||
)
|
||||
@@ -75,7 +76,7 @@ def _real_main(argv=None):
|
||||
|
||||
# Dump user agent
|
||||
if opts.dump_user_agent:
|
||||
write_string(std_headers['User-Agent'] + '\n', out=sys.stdout)
|
||||
compat_print(std_headers['User-Agent'])
|
||||
sys.exit(0)
|
||||
|
||||
# Batch file verification
|
||||
@@ -100,10 +101,10 @@ def _real_main(argv=None):
|
||||
|
||||
if opts.list_extractors:
|
||||
for ie in list_extractors(opts.age_limit):
|
||||
write_string(ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie._WORKING else '') + '\n', out=sys.stdout)
|
||||
compat_print(ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie._WORKING else ''))
|
||||
matchedUrls = [url for url in all_urls if ie.suitable(url)]
|
||||
for mu in matchedUrls:
|
||||
write_string(' ' + mu + '\n', out=sys.stdout)
|
||||
compat_print(' ' + mu)
|
||||
sys.exit(0)
|
||||
if opts.list_extractor_descriptions:
|
||||
for ie in list_extractors(opts.age_limit):
|
||||
@@ -116,7 +117,7 @@ def _real_main(argv=None):
|
||||
_SEARCHES = ('cute kittens', 'slithering pythons', 'falling cat', 'angry poodle', 'purple fish', 'running tortoise', 'sleeping bunny', 'burping cow')
|
||||
_COUNTS = ('', '5', '10', 'all')
|
||||
desc += ' (Example: "%s%s:%s" )' % (ie.SEARCH_KEY, random.choice(_COUNTS), random.choice(_SEARCHES))
|
||||
write_string(desc + '\n', out=sys.stdout)
|
||||
compat_print(desc)
|
||||
sys.exit(0)
|
||||
|
||||
# Conflicting, missing and erroneous options
|
||||
|
||||
2246
youtube_dl/compat.py
2246
youtube_dl/compat.py
File diff suppressed because it is too large
Load Diff
@@ -85,7 +85,7 @@ class ExternalFD(FileDownloader):
|
||||
cmd, stderr=subprocess.PIPE)
|
||||
_, stderr = p.communicate()
|
||||
if p.returncode != 0:
|
||||
self.to_stderr(stderr.decode('utf-8', 'replace'))
|
||||
self.to_stderr(stderr)
|
||||
return p.returncode
|
||||
|
||||
|
||||
@@ -210,7 +210,6 @@ class FFmpegFD(ExternalFD):
|
||||
# args += ['-http_proxy', proxy]
|
||||
env = os.environ.copy()
|
||||
compat_setenv('HTTP_PROXY', proxy, env=env)
|
||||
compat_setenv('http_proxy', proxy, env=env)
|
||||
|
||||
protocol = info_dict.get('protocol')
|
||||
|
||||
|
||||
@@ -23,17 +23,11 @@ class HlsFD(FragmentFD):
|
||||
UNSUPPORTED_FEATURES = (
|
||||
r'#EXT-X-KEY:METHOD=(?!NONE)', # encrypted streams [1]
|
||||
r'#EXT-X-BYTERANGE', # playlists composed of byte ranges of media files [2]
|
||||
|
||||
# Live streams heuristic does not always work (e.g. geo restricted to Germany
|
||||
# http://hls-geo.daserste.de/i/videoportal/Film/c_620000/622873/format,716451,716457,716450,716458,716459,.mp4.csmil/index_4_av.m3u8?null=0)
|
||||
# r'#EXT-X-MEDIA-SEQUENCE:(?!0$)', # live streams [3]
|
||||
|
||||
# This heuristic also is not correct since segments may not be appended as well.
|
||||
# Twitch vods of finished streams have EXT-X-PLAYLIST-TYPE:EVENT despite
|
||||
# no segments will definitely be appended to the end of the playlist.
|
||||
# r'#EXT-X-PLAYLIST-TYPE:EVENT', # media segments may be appended to the end of
|
||||
# # event media playlists [4]
|
||||
|
||||
r'#EXT-X-PLAYLIST-TYPE:EVENT', # media segments may be appended to the end of
|
||||
# event media playlists [4]
|
||||
# 1. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.4
|
||||
# 2. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.2
|
||||
# 3. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.2
|
||||
|
||||
@@ -156,10 +156,7 @@ class AdobeTVVideoIE(InfoExtractor):
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
video_data = self._parse_json(self._search_regex(
|
||||
r'var\s+bridge\s*=\s*([^;]+);', webpage, 'bridged data'), video_id)
|
||||
video_data = self._download_json(url + '?format=json', video_id)
|
||||
|
||||
formats = [{
|
||||
'format_id': '%s-%s' % (determine_ext(source['src']), source.get('height')),
|
||||
|
||||
@@ -1,133 +0,0 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..compat import (
|
||||
compat_urllib_parse_urlparse,
|
||||
compat_urlparse,
|
||||
)
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
int_or_none,
|
||||
xpath_element,
|
||||
xpath_text,
|
||||
)
|
||||
|
||||
|
||||
class AfreecaTVIE(InfoExtractor):
|
||||
IE_DESC = 'afreecatv.com'
|
||||
_VALID_URL = r'''(?x)^
|
||||
https?://(?:(live|afbbs|www)\.)?afreeca(?:tv)?\.com(?::\d+)?
|
||||
(?:
|
||||
/app/(?:index|read_ucc_bbs)\.cgi|
|
||||
/player/[Pp]layer\.(?:swf|html))
|
||||
\?.*?\bnTitleNo=(?P<id>\d+)'''
|
||||
_TESTS = [{
|
||||
'url': 'http://live.afreecatv.com:8079/app/index.cgi?szType=read_ucc_bbs&szBjId=dailyapril&nStationNo=16711924&nBbsNo=18605867&nTitleNo=36164052&szSkin=',
|
||||
'md5': 'f72c89fe7ecc14c1b5ce506c4996046e',
|
||||
'info_dict': {
|
||||
'id': '36164052',
|
||||
'ext': 'mp4',
|
||||
'title': '데일리 에이프릴 요정들의 시상식!',
|
||||
'thumbnail': 're:^https?://(?:video|st)img.afreecatv.com/.*$',
|
||||
'uploader': 'dailyapril',
|
||||
'uploader_id': 'dailyapril',
|
||||
'upload_date': '20160503',
|
||||
}
|
||||
}, {
|
||||
'url': 'http://afbbs.afreecatv.com:8080/app/read_ucc_bbs.cgi?nStationNo=16711924&nTitleNo=36153164&szBjId=dailyapril&nBbsNo=18605867',
|
||||
'info_dict': {
|
||||
'id': '36153164',
|
||||
'title': "BJ유트루와 함께하는 '팅커벨 메이크업!'",
|
||||
'thumbnail': 're:^https?://(?:video|st)img.afreecatv.com/.*$',
|
||||
'uploader': 'dailyapril',
|
||||
'uploader_id': 'dailyapril',
|
||||
},
|
||||
'playlist_count': 2,
|
||||
'playlist': [{
|
||||
'md5': 'd8b7c174568da61d774ef0203159bf97',
|
||||
'info_dict': {
|
||||
'id': '36153164_1',
|
||||
'ext': 'mp4',
|
||||
'title': "BJ유트루와 함께하는 '팅커벨 메이크업!'",
|
||||
'upload_date': '20160502',
|
||||
},
|
||||
}, {
|
||||
'md5': '58f2ce7f6044e34439ab2d50612ab02b',
|
||||
'info_dict': {
|
||||
'id': '36153164_2',
|
||||
'ext': 'mp4',
|
||||
'title': "BJ유트루와 함께하는 '팅커벨 메이크업!'",
|
||||
'upload_date': '20160502',
|
||||
},
|
||||
}],
|
||||
}, {
|
||||
'url': 'http://www.afreecatv.com/player/Player.swf?szType=szBjId=djleegoon&nStationNo=11273158&nBbsNo=13161095&nTitleNo=36327652',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
@staticmethod
|
||||
def parse_video_key(key):
|
||||
video_key = {}
|
||||
m = re.match(r'^(?P<upload_date>\d{8})_\w+_(?P<part>\d+)$', key)
|
||||
if m:
|
||||
video_key['upload_date'] = m.group('upload_date')
|
||||
video_key['part'] = m.group('part')
|
||||
return video_key
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
parsed_url = compat_urllib_parse_urlparse(url)
|
||||
info_url = compat_urlparse.urlunparse(parsed_url._replace(
|
||||
netloc='afbbs.afreecatv.com:8080',
|
||||
path='/api/video/get_video_info.php'))
|
||||
video_xml = self._download_xml(info_url, video_id)
|
||||
|
||||
if xpath_element(video_xml, './track/video/file') is None:
|
||||
raise ExtractorError('Specified AfreecaTV video does not exist',
|
||||
expected=True)
|
||||
|
||||
title = xpath_text(video_xml, './track/title', 'title')
|
||||
uploader = xpath_text(video_xml, './track/nickname', 'uploader')
|
||||
uploader_id = xpath_text(video_xml, './track/bj_id', 'uploader id')
|
||||
duration = int_or_none(xpath_text(video_xml, './track/duration',
|
||||
'duration'))
|
||||
thumbnail = xpath_text(video_xml, './track/titleImage', 'thumbnail')
|
||||
|
||||
entries = []
|
||||
for i, video_file in enumerate(video_xml.findall('./track/video/file')):
|
||||
video_key = self.parse_video_key(video_file.get('key', ''))
|
||||
if not video_key:
|
||||
continue
|
||||
entries.append({
|
||||
'id': '%s_%s' % (video_id, video_key.get('part', i + 1)),
|
||||
'title': title,
|
||||
'upload_date': video_key.get('upload_date'),
|
||||
'duration': int_or_none(video_file.get('duration')),
|
||||
'url': video_file.text,
|
||||
})
|
||||
|
||||
info = {
|
||||
'id': video_id,
|
||||
'title': title,
|
||||
'uploader': uploader,
|
||||
'uploader_id': uploader_id,
|
||||
'duration': duration,
|
||||
'thumbnail': thumbnail,
|
||||
}
|
||||
|
||||
if len(entries) > 1:
|
||||
info['_type'] = 'multi_video'
|
||||
info['entries'] = entries
|
||||
elif len(entries) == 1:
|
||||
info['url'] = entries[0]['url']
|
||||
info['upload_date'] = entries[0].get('upload_date')
|
||||
else:
|
||||
raise ExtractorError(
|
||||
'No files found for the specified AfreecaTV video, either'
|
||||
' the URL is incorrect or the video has been made private.',
|
||||
expected=True)
|
||||
|
||||
return info
|
||||
@@ -24,10 +24,10 @@ class AftonbladetIE(InfoExtractor):
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
# find internal video meta data
|
||||
meta_url = 'http://aftonbladet-play-metadata.cdn.drvideo.aptoma.no/video/%s.json'
|
||||
meta_url = 'http://aftonbladet-play.drlib.aptoma.no/video/%s.json'
|
||||
player_config = self._parse_json(self._html_search_regex(
|
||||
r'data-player-config="([^"]+)"', webpage, 'player config'), video_id)
|
||||
internal_meta_id = player_config['aptomaVideoId']
|
||||
internal_meta_id = player_config['videoId']
|
||||
internal_meta_url = meta_url % internal_meta_id
|
||||
internal_meta_json = self._download_json(
|
||||
internal_meta_url, video_id, 'Downloading video meta data')
|
||||
|
||||
@@ -8,6 +8,7 @@ from .generic import GenericIE
|
||||
from ..utils import (
|
||||
determine_ext,
|
||||
ExtractorError,
|
||||
get_element_by_attribute,
|
||||
qualities,
|
||||
int_or_none,
|
||||
parse_duration,
|
||||
@@ -273,3 +274,41 @@ class ARDIE(InfoExtractor):
|
||||
'upload_date': upload_date,
|
||||
'thumbnail': thumbnail,
|
||||
}
|
||||
|
||||
|
||||
class SportschauIE(ARDMediathekIE):
|
||||
IE_NAME = 'Sportschau'
|
||||
_VALID_URL = r'(?P<baseurl>https?://(?:www\.)?sportschau\.de/(?:[^/]+/)+video(?P<id>[^/#?]+))\.html'
|
||||
_TESTS = [{
|
||||
'url': 'http://www.sportschau.de/tourdefrance/videoseppeltkokainhatnichtsmitklassischemdopingzutun100.html',
|
||||
'info_dict': {
|
||||
'id': 'seppeltkokainhatnichtsmitklassischemdopingzutun100',
|
||||
'ext': 'mp4',
|
||||
'title': 'Seppelt: "Kokain hat nichts mit klassischem Doping zu tun"',
|
||||
'thumbnail': 're:^https?://.*\.jpg$',
|
||||
'description': 'Der ARD-Doping Experte Hajo Seppelt gibt seine Einschätzung zum ersten Dopingfall der diesjährigen Tour de France um den Italiener Luca Paolini ab.',
|
||||
},
|
||||
'params': {
|
||||
# m3u8 download
|
||||
'skip_download': True,
|
||||
},
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
video_id = mobj.group('id')
|
||||
base_url = mobj.group('baseurl')
|
||||
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
title = get_element_by_attribute('class', 'headline', webpage)
|
||||
description = self._html_search_meta('description', webpage, 'description')
|
||||
|
||||
info = self._extract_media_info(
|
||||
base_url + '-mc_defaultQuality-h.json', webpage, video_id)
|
||||
|
||||
info.update({
|
||||
'title': title,
|
||||
'description': description,
|
||||
})
|
||||
|
||||
return info
|
||||
|
||||
@@ -180,14 +180,11 @@ class ArteTVBaseIE(InfoExtractor):
|
||||
|
||||
class ArteTVPlus7IE(ArteTVBaseIE):
|
||||
IE_NAME = 'arte.tv:+7'
|
||||
_VALID_URL = r'https?://(?:(?:www|sites)\.)?arte\.tv/[^/]+/(?P<lang>fr|de|en|es)/(?:[^/]+/)*(?P<id>[^/?#&]+)'
|
||||
_VALID_URL = r'https?://(?:www\.)?arte\.tv/guide/(?P<lang>fr|de|en|es)/(?:(?:sendungen|emissions|embed)/)?(?P<id>[^/]+)/(?P<name>[^/?#&]+)'
|
||||
|
||||
_TESTS = [{
|
||||
'url': 'http://www.arte.tv/guide/de/sendungen/XEN/xenius/?vid=055918-015_PLUS7-D',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://sites.arte.tv/karambolage/de/video/karambolage-22',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
@classmethod
|
||||
@@ -243,10 +240,10 @@ class ArteTVPlus7IE(ArteTVBaseIE):
|
||||
return self._extract_from_json_url(json_url, video_id, lang, title=title)
|
||||
# Different kind of embed URL (e.g.
|
||||
# http://www.arte.tv/magazine/trepalium/fr/episode-0406-replay-trepalium)
|
||||
entries = [
|
||||
self.url_result(url)
|
||||
for _, url in re.findall(r'<iframe[^>]+src=(["\'])(?P<url>.+?)\1', webpage)]
|
||||
return self.playlist_result(entries)
|
||||
embed_url = self._search_regex(
|
||||
r'<iframe[^>]+src=(["\'])(?P<url>.+?)\1',
|
||||
webpage, 'embed url', group='url')
|
||||
return self.url_result(embed_url)
|
||||
|
||||
|
||||
# It also uses the arte_vp_url url from the webpage to extract the information
|
||||
@@ -255,17 +252,22 @@ class ArteTVCreativeIE(ArteTVPlus7IE):
|
||||
_VALID_URL = r'https?://creative\.arte\.tv/(?P<lang>fr|de|en|es)/(?:[^/]+/)*(?P<id>[^/?#&]+)'
|
||||
|
||||
_TESTS = [{
|
||||
'url': 'http://creative.arte.tv/fr/episode/osmosis-episode-1',
|
||||
'url': 'http://creative.arte.tv/de/magazin/agentur-amateur-corporate-design',
|
||||
'info_dict': {
|
||||
'id': '057405-001-A',
|
||||
'id': '72176',
|
||||
'ext': 'mp4',
|
||||
'title': 'OSMOSIS - N\'AYEZ PLUS PEUR D\'AIMER (1)',
|
||||
'upload_date': '20150716',
|
||||
'title': 'Folge 2 - Corporate Design',
|
||||
'upload_date': '20131004',
|
||||
},
|
||||
}, {
|
||||
'url': 'http://creative.arte.tv/fr/Monty-Python-Reunion',
|
||||
'playlist_count': 11,
|
||||
'add_ie': ['Youtube'],
|
||||
'info_dict': {
|
||||
'id': '160676',
|
||||
'ext': 'mp4',
|
||||
'title': 'Monty Python live (mostly)',
|
||||
'description': 'Événement ! Quarante-cinq ans après leurs premiers succès, les légendaires Monty Python remontent sur scène.\n',
|
||||
'upload_date': '20140805',
|
||||
}
|
||||
}, {
|
||||
'url': 'http://creative.arte.tv/de/episode/agentur-amateur-4-der-erste-kunde',
|
||||
'only_matching': True,
|
||||
@@ -347,13 +349,14 @@ class ArteTVCinemaIE(ArteTVPlus7IE):
|
||||
_VALID_URL = r'https?://cinema\.arte\.tv/(?P<lang>fr|de|en|es)/(?P<id>.+)'
|
||||
|
||||
_TESTS = [{
|
||||
'url': 'http://cinema.arte.tv/fr/article/les-ailes-du-desir-de-julia-reck',
|
||||
'md5': 'a5b9dd5575a11d93daf0e3f404f45438',
|
||||
'url': 'http://cinema.arte.tv/de/node/38291',
|
||||
'md5': '6b275511a5107c60bacbeeda368c3aa1',
|
||||
'info_dict': {
|
||||
'id': '062494-000-A',
|
||||
'id': '055876-000_PWA12025-D',
|
||||
'ext': 'mp4',
|
||||
'title': 'Film lauréat du concours web - "Les ailes du désir" de Julia Reck',
|
||||
'upload_date': '20150807',
|
||||
'title': 'Tod auf dem Nil',
|
||||
'upload_date': '20160122',
|
||||
'description': 'md5:7f749bbb77d800ef2be11d54529b96bc',
|
||||
},
|
||||
}]
|
||||
|
||||
|
||||
@@ -6,7 +6,6 @@ import time
|
||||
|
||||
from .common import InfoExtractor
|
||||
from .soundcloud import SoundcloudIE
|
||||
from ..compat import compat_str
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
url_basename,
|
||||
@@ -137,7 +136,7 @@ class AudiomackAlbumIE(InfoExtractor):
|
||||
result[resultkey] = api_response[apikey]
|
||||
song_id = url_basename(api_response['url']).rpartition('.')[0]
|
||||
result['entries'].append({
|
||||
'id': compat_str(api_response.get('id', song_id)),
|
||||
'id': api_response.get('id', song_id),
|
||||
'uploader': api_response.get('artist'),
|
||||
'title': api_response.get('title', song_id),
|
||||
'url': api_response['url'],
|
||||
|
||||
@@ -46,7 +46,6 @@ class AzubuIE(InfoExtractor):
|
||||
'uploader_id': 272749,
|
||||
'view_count': int,
|
||||
},
|
||||
'skip': 'Channel offline',
|
||||
},
|
||||
]
|
||||
|
||||
@@ -57,26 +56,22 @@ class AzubuIE(InfoExtractor):
|
||||
'http://www.azubu.tv/api/video/%s' % video_id, video_id)['data']
|
||||
|
||||
title = data['title'].strip()
|
||||
description = data.get('description')
|
||||
thumbnail = data.get('thumbnail')
|
||||
view_count = data.get('view_count')
|
||||
user = data.get('user', {})
|
||||
uploader = user.get('username')
|
||||
uploader_id = user.get('id')
|
||||
description = data['description']
|
||||
thumbnail = data['thumbnail']
|
||||
view_count = data['view_count']
|
||||
uploader = data['user']['username']
|
||||
uploader_id = data['user']['id']
|
||||
|
||||
stream_params = json.loads(data['stream_params'])
|
||||
|
||||
timestamp = float_or_none(stream_params.get('creationDate'), 1000)
|
||||
duration = float_or_none(stream_params.get('length'), 1000)
|
||||
timestamp = float_or_none(stream_params['creationDate'], 1000)
|
||||
duration = float_or_none(stream_params['length'], 1000)
|
||||
|
||||
renditions = stream_params.get('renditions') or []
|
||||
video = stream_params.get('FLVFullLength') or stream_params.get('videoFullLength')
|
||||
if video:
|
||||
renditions.append(video)
|
||||
|
||||
if not renditions and not user.get('channel', {}).get('is_live', True):
|
||||
raise ExtractorError('%s said: channel is offline.' % self.IE_NAME, expected=True)
|
||||
|
||||
formats = [{
|
||||
'url': fmt['url'],
|
||||
'width': fmt['frameWidth'],
|
||||
|
||||
@@ -31,7 +31,7 @@ class BBCCoUkIE(InfoExtractor):
|
||||
music/clips[/#]|
|
||||
radio/player/
|
||||
)
|
||||
(?P<id>%s)(?!/(?:episodes|broadcasts|clips))
|
||||
(?P<id>%s)
|
||||
''' % _ID_REGEX
|
||||
|
||||
_MEDIASELECTOR_URLS = [
|
||||
@@ -192,7 +192,6 @@ class BBCCoUkIE(InfoExtractor):
|
||||
# rtmp download
|
||||
'skip_download': True,
|
||||
},
|
||||
'skip': 'Now it\'s really geo-restricted',
|
||||
}, {
|
||||
# compact player (https://github.com/rg3/youtube-dl/issues/8147)
|
||||
'url': 'http://www.bbc.co.uk/programmes/p028bfkf/player',
|
||||
@@ -699,9 +698,7 @@ class BBCIE(BBCCoUkIE):
|
||||
|
||||
@classmethod
|
||||
def suitable(cls, url):
|
||||
EXCLUDE_IE = (BBCCoUkIE, BBCCoUkArticleIE, BBCCoUkIPlayerPlaylistIE, BBCCoUkPlaylistIE)
|
||||
return (False if any(ie.suitable(url) for ie in EXCLUDE_IE)
|
||||
else super(BBCIE, cls).suitable(url))
|
||||
return False if BBCCoUkIE.suitable(url) or BBCCoUkArticleIE.suitable(url) else super(BBCIE, cls).suitable(url)
|
||||
|
||||
def _extract_from_media_meta(self, media_meta, video_id):
|
||||
# Direct links to media in media metadata (e.g.
|
||||
@@ -978,72 +975,3 @@ class BBCCoUkArticleIE(InfoExtractor):
|
||||
r'<div[^>]+typeof="Clip"[^>]+resource="([^"]+)"', webpage)]
|
||||
|
||||
return self.playlist_result(entries, playlist_id, title, description)
|
||||
|
||||
|
||||
class BBCCoUkPlaylistBaseIE(InfoExtractor):
|
||||
def _real_extract(self, url):
|
||||
playlist_id = self._match_id(url)
|
||||
|
||||
webpage = self._download_webpage(url, playlist_id)
|
||||
|
||||
entries = [
|
||||
self.url_result(self._URL_TEMPLATE % video_id, BBCCoUkIE.ie_key())
|
||||
for video_id in re.findall(
|
||||
self._VIDEO_ID_TEMPLATE % BBCCoUkIE._ID_REGEX, webpage)]
|
||||
|
||||
title, description = self._extract_title_and_description(webpage)
|
||||
|
||||
return self.playlist_result(entries, playlist_id, title, description)
|
||||
|
||||
|
||||
class BBCCoUkIPlayerPlaylistIE(BBCCoUkPlaylistBaseIE):
|
||||
IE_NAME = 'bbc.co.uk:iplayer:playlist'
|
||||
_VALID_URL = r'https?://(?:www\.)?bbc\.co\.uk/iplayer/episodes/(?P<id>%s)' % BBCCoUkIE._ID_REGEX
|
||||
_URL_TEMPLATE = 'http://www.bbc.co.uk/iplayer/episode/%s'
|
||||
_VIDEO_ID_TEMPLATE = r'data-ip-id=["\'](%s)'
|
||||
_TEST = {
|
||||
'url': 'http://www.bbc.co.uk/iplayer/episodes/b05rcz9v',
|
||||
'info_dict': {
|
||||
'id': 'b05rcz9v',
|
||||
'title': 'The Disappearance',
|
||||
'description': 'French thriller serial about a missing teenager.',
|
||||
},
|
||||
'playlist_mincount': 6,
|
||||
}
|
||||
|
||||
def _extract_title_and_description(self, webpage):
|
||||
title = self._search_regex(r'<h1>([^<]+)</h1>', webpage, 'title', fatal=False)
|
||||
description = self._search_regex(
|
||||
r'<p[^>]+class=(["\'])subtitle\1[^>]*>(?P<value>[^<]+)</p>',
|
||||
webpage, 'description', fatal=False, group='value')
|
||||
return title, description
|
||||
|
||||
|
||||
class BBCCoUkPlaylistIE(BBCCoUkPlaylistBaseIE):
|
||||
IE_NAME = 'bbc.co.uk:playlist'
|
||||
_VALID_URL = r'https?://(?:www\.)?bbc\.co\.uk/programmes/(?P<id>%s)/(?:episodes|broadcasts|clips)' % BBCCoUkIE._ID_REGEX
|
||||
_URL_TEMPLATE = 'http://www.bbc.co.uk/programmes/%s'
|
||||
_VIDEO_ID_TEMPLATE = r'data-pid=["\'](%s)'
|
||||
_TESTS = [{
|
||||
'url': 'http://www.bbc.co.uk/programmes/b05rcz9v/clips',
|
||||
'info_dict': {
|
||||
'id': 'b05rcz9v',
|
||||
'title': 'The Disappearance - Clips - BBC Four',
|
||||
'description': 'French thriller serial about a missing teenager.',
|
||||
},
|
||||
'playlist_mincount': 7,
|
||||
}, {
|
||||
'url': 'http://www.bbc.co.uk/programmes/b05rcz9v/broadcasts/2016/06',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://www.bbc.co.uk/programmes/b05rcz9v/clips',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://www.bbc.co.uk/programmes/b055jkys/episodes/player',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _extract_title_and_description(self, webpage):
|
||||
title = self._og_search_title(webpage, fatal=False)
|
||||
description = self._og_search_description(webpage)
|
||||
return title, description
|
||||
|
||||
@@ -1,27 +1,31 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
from .mtv import MTVServicesInfoExtractor
|
||||
from ..utils import unified_strdate
|
||||
from ..compat import compat_urllib_parse_urlencode
|
||||
from .common import InfoExtractor
|
||||
from ..compat import compat_urllib_parse_unquote
|
||||
from ..utils import (
|
||||
xpath_text,
|
||||
xpath_with_ns,
|
||||
int_or_none,
|
||||
parse_iso8601,
|
||||
)
|
||||
|
||||
|
||||
class BetIE(MTVServicesInfoExtractor):
|
||||
class BetIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?bet\.com/(?:[^/]+/)+(?P<id>.+?)\.html'
|
||||
_TESTS = [
|
||||
{
|
||||
'url': 'http://www.bet.com/news/politics/2014/12/08/in-bet-exclusive-obama-talks-race-and-racism.html',
|
||||
'info_dict': {
|
||||
'id': '07e96bd3-8850-3051-b856-271b457f0ab8',
|
||||
'id': 'news/national/2014/a-conversation-with-president-obama',
|
||||
'display_id': 'in-bet-exclusive-obama-talks-race-and-racism',
|
||||
'ext': 'flv',
|
||||
'title': 'A Conversation With President Obama',
|
||||
'description': 'President Obama urges persistence in confronting racism and bias.',
|
||||
'description': 'md5:699d0652a350cf3e491cd15cc745b5da',
|
||||
'duration': 1534,
|
||||
'timestamp': 1418075340,
|
||||
'upload_date': '20141208',
|
||||
'uploader': 'admin',
|
||||
'thumbnail': 're:(?i)^https?://.*\.jpg$',
|
||||
'subtitles': {
|
||||
'en': 'mincount:2',
|
||||
}
|
||||
},
|
||||
'params': {
|
||||
# rtmp download
|
||||
@@ -31,17 +35,16 @@ class BetIE(MTVServicesInfoExtractor):
|
||||
{
|
||||
'url': 'http://www.bet.com/video/news/national/2014/justice-for-ferguson-a-community-reacts.html',
|
||||
'info_dict': {
|
||||
'id': '9f516bf1-7543-39c4-8076-dd441b459ba9',
|
||||
'id': 'news/national/2014/justice-for-ferguson-a-community-reacts',
|
||||
'display_id': 'justice-for-ferguson-a-community-reacts',
|
||||
'ext': 'flv',
|
||||
'title': 'Justice for Ferguson: A Community Reacts',
|
||||
'description': 'A BET News special.',
|
||||
'duration': 1696,
|
||||
'timestamp': 1416942360,
|
||||
'upload_date': '20141125',
|
||||
'uploader': 'admin',
|
||||
'thumbnail': 're:(?i)^https?://.*\.jpg$',
|
||||
'subtitles': {
|
||||
'en': 'mincount:2',
|
||||
}
|
||||
},
|
||||
'params': {
|
||||
# rtmp download
|
||||
@@ -50,32 +53,57 @@ class BetIE(MTVServicesInfoExtractor):
|
||||
}
|
||||
]
|
||||
|
||||
_FEED_URL = "http://feeds.mtvnservices.com/od/feed/bet-mrss-player"
|
||||
|
||||
def _get_feed_query(self, uri):
|
||||
return compat_urllib_parse_urlencode({
|
||||
'uuid': uri,
|
||||
})
|
||||
|
||||
def _extract_mgid(self, webpage):
|
||||
return self._search_regex(r'data-uri="([^"]+)', webpage, 'mgid')
|
||||
|
||||
def _real_extract(self, url):
|
||||
display_id = self._match_id(url)
|
||||
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
mgid = self._extract_mgid(webpage)
|
||||
videos_info = self._get_videos_info(mgid)
|
||||
|
||||
info_dict = videos_info['entries'][0]
|
||||
media_url = compat_urllib_parse_unquote(self._search_regex(
|
||||
[r'mediaURL\s*:\s*"([^"]+)"', r"var\s+mrssMediaUrl\s*=\s*'([^']+)'"],
|
||||
webpage, 'media URL'))
|
||||
|
||||
upload_date = unified_strdate(self._html_search_meta('date', webpage))
|
||||
description = self._html_search_meta('description', webpage)
|
||||
video_id = self._search_regex(
|
||||
r'/video/(.*)/_jcr_content/', media_url, 'video id')
|
||||
|
||||
info_dict.update({
|
||||
mrss = self._download_xml(media_url, display_id)
|
||||
|
||||
item = mrss.find('./channel/item')
|
||||
|
||||
NS_MAP = {
|
||||
'dc': 'http://purl.org/dc/elements/1.1/',
|
||||
'media': 'http://search.yahoo.com/mrss/',
|
||||
'ka': 'http://kickapps.com/karss',
|
||||
}
|
||||
|
||||
title = xpath_text(item, './title', 'title')
|
||||
description = xpath_text(
|
||||
item, './description', 'description', fatal=False)
|
||||
|
||||
timestamp = parse_iso8601(xpath_text(
|
||||
item, xpath_with_ns('./dc:date', NS_MAP),
|
||||
'upload date', fatal=False))
|
||||
uploader = xpath_text(
|
||||
item, xpath_with_ns('./dc:creator', NS_MAP),
|
||||
'uploader', fatal=False)
|
||||
|
||||
media_content = item.find(
|
||||
xpath_with_ns('./media:content', NS_MAP))
|
||||
duration = int_or_none(media_content.get('duration'))
|
||||
smil_url = media_content.get('url')
|
||||
|
||||
thumbnail = media_content.find(
|
||||
xpath_with_ns('./media:thumbnail', NS_MAP)).get('url')
|
||||
|
||||
formats = self._extract_smil_formats(smil_url, display_id)
|
||||
self._sort_formats(formats)
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'display_id': display_id,
|
||||
'title': title,
|
||||
'description': description,
|
||||
'upload_date': upload_date,
|
||||
})
|
||||
|
||||
return info_dict
|
||||
'thumbnail': thumbnail,
|
||||
'timestamp': timestamp,
|
||||
'uploader': uploader,
|
||||
'duration': duration,
|
||||
'formats': formats,
|
||||
}
|
||||
|
||||
@@ -102,22 +102,6 @@ class BiliBiliIE(InfoExtractor):
|
||||
'uploader_id': '151066',
|
||||
},
|
||||
}],
|
||||
}, {
|
||||
# Missing upload time
|
||||
'url': 'http://www.bilibili.com/video/av1867637/',
|
||||
'info_dict': {
|
||||
'id': '2880301',
|
||||
'ext': 'flv',
|
||||
'title': '【HDTV】【喜剧】岳父岳母真难当 （2014）【法国票房冠军】',
|
||||
'description': '一个信奉天主教的法国旧式传统资产阶级家庭中有四个女儿。三个女儿却分别找了阿拉伯、犹太、中国丈夫，老夫老妻唯独期盼剩下未嫁的小女儿能找一个信奉天主教的法国白人，结果没想到小女儿找了一位非裔黑人……【这次应该不会跳帧了】',
|
||||
'uploader': '黑夜为猫',
|
||||
'uploader_id': '610729',
|
||||
},
|
||||
'params': {
|
||||
# Just to test metadata extraction
|
||||
'skip_download': True,
|
||||
},
|
||||
'expected_warnings': ['upload time'],
|
||||
}]
|
||||
|
||||
# BiliBili blocks keys from time to time. The current key is extracted from
|
||||
@@ -188,7 +172,6 @@ class BiliBiliIE(InfoExtractor):
|
||||
description = self._html_search_meta('description', webpage)
|
||||
datetime_str = self._html_search_regex(
|
||||
r'<time[^>]+datetime="([^"]+)"', webpage, 'upload time', fatal=False)
|
||||
timestamp = None
|
||||
if datetime_str:
|
||||
timestamp = calendar.timegm(datetime.datetime.strptime(datetime_str, '%Y-%m-%dT%H:%M').timetuple())
|
||||
|
||||
|
||||
@@ -29,8 +29,7 @@ class BRIE(InfoExtractor):
|
||||
'duration': 180,
|
||||
'uploader': 'Reinhard Weber',
|
||||
'upload_date': '20150422',
|
||||
},
|
||||
'skip': '404 not found',
|
||||
}
|
||||
},
|
||||
{
|
||||
'url': 'http://www.br.de/nachrichten/oberbayern/inhalt/muenchner-polizeipraesident-schreiber-gestorben-100.html',
|
||||
@@ -41,8 +40,7 @@ class BRIE(InfoExtractor):
|
||||
'title': 'Manfred Schreiber ist tot',
|
||||
'description': 'md5:b454d867f2a9fc524ebe88c3f5092d97',
|
||||
'duration': 26,
|
||||
},
|
||||
'skip': '404 not found',
|
||||
}
|
||||
},
|
||||
{
|
||||
'url': 'https://www.br-klassik.de/audio/peeping-tom-premierenkritik-dance-festival-muenchen-100.html',
|
||||
@@ -53,8 +51,7 @@ class BRIE(InfoExtractor):
|
||||
'title': 'Kurzweilig und sehr bewegend',
|
||||
'description': 'md5:0351996e3283d64adeb38ede91fac54e',
|
||||
'duration': 296,
|
||||
},
|
||||
'skip': '404 not found',
|
||||
}
|
||||
},
|
||||
{
|
||||
'url': 'http://www.br.de/radio/bayern1/service/team/videos/team-video-erdelt100.html',
|
||||
|
||||
@@ -4,11 +4,11 @@ from __future__ import unicode_literals
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..compat import compat_urllib_parse_urlparse
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
HEADRequest,
|
||||
unified_strdate,
|
||||
url_basename,
|
||||
qualities,
|
||||
int_or_none,
|
||||
)
|
||||
@@ -16,38 +16,24 @@ from ..utils import (
|
||||
|
||||
class CanalplusIE(InfoExtractor):
|
||||
IE_DESC = 'canalplus.fr, piwiplus.fr and d8.tv'
|
||||
_VALID_URL = r'''(?x)
|
||||
https?://
|
||||
(?:
|
||||
(?:
|
||||
(?:(?:www|m)\.)?canalplus\.fr|
|
||||
(?:www\.)?piwiplus\.fr|
|
||||
(?:www\.)?d8\.tv|
|
||||
(?:www\.)?d17\.tv|
|
||||
(?:www\.)?itele\.fr
|
||||
)/(?:(?:[^/]+/)*(?P<display_id>[^/?#&]+))?(?:\?.*\bvid=(?P<vid>\d+))?|
|
||||
player\.canalplus\.fr/#/(?P<id>\d+)
|
||||
)
|
||||
|
||||
'''
|
||||
_VALID_URL = r'https?://(?:www\.(?P<site>canalplus\.fr|piwiplus\.fr|d8\.tv|itele\.fr)/.*?/(?P<path>.*)|player\.canalplus\.fr/#/(?P<id>[0-9]+))'
|
||||
_VIDEO_INFO_TEMPLATE = 'http://service.canal-plus.com/video/rest/getVideosLiees/%s/%s?format=json'
|
||||
_SITE_ID_MAP = {
|
||||
'canalplus': 'cplus',
|
||||
'piwiplus': 'teletoon',
|
||||
'd8': 'd8',
|
||||
'd17': 'd17',
|
||||
'itele': 'itele',
|
||||
'canalplus.fr': 'cplus',
|
||||
'piwiplus.fr': 'teletoon',
|
||||
'd8.tv': 'd8',
|
||||
'itele.fr': 'itele',
|
||||
}
|
||||
|
||||
_TESTS = [{
|
||||
'url': 'http://www.canalplus.fr/c-emissions/pid1830-c-zapping.html?vid=1192814',
|
||||
'md5': '41f438a4904f7664b91b4ed0dec969dc',
|
||||
'url': 'http://www.canalplus.fr/c-emissions/pid1830-c-zapping.html?vid=1263092',
|
||||
'md5': '12164a6f14ff6df8bd628e8ba9b10b78',
|
||||
'info_dict': {
|
||||
'id': '1192814',
|
||||
'id': '1263092',
|
||||
'ext': 'mp4',
|
||||
'title': "L'Année du Zapping 2014 - L'Année du Zapping 2014",
|
||||
'description': "Toute l'année 2014 dans un Zapping exceptionnel !",
|
||||
'upload_date': '20150105',
|
||||
'title': 'Le Zapping - 13/05/15',
|
||||
'description': 'md5:09738c0d06be4b5d06a0940edb0da73f',
|
||||
'upload_date': '20150513',
|
||||
},
|
||||
}, {
|
||||
'url': 'http://www.piwiplus.fr/videos-piwi/pid1405-le-labyrinthe-boing-super-ranger.html?vid=1108190',
|
||||
@@ -60,45 +46,35 @@ class CanalplusIE(InfoExtractor):
|
||||
},
|
||||
'skip': 'Only works from France',
|
||||
}, {
|
||||
'url': 'http://www.d8.tv/d8-docs-mags/pid5198-d8-en-quete-d-actualite.html?vid=1390231',
|
||||
'url': 'http://www.d8.tv/d8-docs-mags/pid6589-d8-campagne-intime.html',
|
||||
'info_dict': {
|
||||
'id': '1390231',
|
||||
'ext': 'mp4',
|
||||
'title': "Vacances pas chères : prix discount ou grosses dépenses ? - En quête d'actualité",
|
||||
'description': 'md5:edb6cf1cb4a1e807b5dd089e1ac8bfc6',
|
||||
'upload_date': '20160512',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
'id': '966289',
|
||||
'ext': 'flv',
|
||||
'title': 'Campagne intime - Documentaire exceptionnel',
|
||||
'description': 'md5:d2643b799fb190846ae09c61e59a859f',
|
||||
'upload_date': '20131108',
|
||||
},
|
||||
'skip': 'videos get deleted after a while',
|
||||
}, {
|
||||
'url': 'http://www.itele.fr/chroniques/invite-bruce-toussaint/thierry-solere-nicolas-sarkozy-officialisera-sa-candidature-a-la-primaire-quand-il-le-voudra-167224',
|
||||
'url': 'http://www.itele.fr/france/video/aubervilliers-un-lycee-en-colere-111559',
|
||||
'md5': '38b8f7934def74f0d6f3ba6c036a5f82',
|
||||
'info_dict': {
|
||||
'id': '1398334',
|
||||
'id': '1213714',
|
||||
'ext': 'mp4',
|
||||
'title': "L'invité de Bruce Toussaint du 07/06/2016 - ",
|
||||
'description': 'md5:40ac7c9ad0feaeb6f605bad986f61324',
|
||||
'upload_date': '20160607',
|
||||
'title': 'Aubervilliers : un lycée en colère - Le 11/02/2015 à 06h45',
|
||||
'description': 'md5:8216206ec53426ea6321321f3b3c16db',
|
||||
'upload_date': '20150211',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
'url': 'http://m.canalplus.fr/?vid=1398231',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://www.d17.tv/emissions/pid8303-lolywood.html?vid=1397061',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
video_id = mobj.groupdict().get('id') or mobj.groupdict().get('vid')
|
||||
video_id = mobj.groupdict().get('id')
|
||||
|
||||
site_id = self._SITE_ID_MAP[compat_urllib_parse_urlparse(url).netloc.rsplit('.', 2)[-2]]
|
||||
site_id = self._SITE_ID_MAP[mobj.group('site') or 'canal']
|
||||
|
||||
# Beware, some subclasses do not define an id group
|
||||
display_id = mobj.group('display_id') or video_id
|
||||
display_id = url_basename(mobj.group('path'))
|
||||
|
||||
if video_id is None:
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
|
||||
@@ -1,88 +0,0 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..compat import compat_str
|
||||
from ..utils import (
|
||||
float_or_none,
|
||||
int_or_none,
|
||||
try_get,
|
||||
)
|
||||
|
||||
|
||||
class CarambaTVIE(InfoExtractor):
|
||||
_VALID_URL = r'(?:carambatv:|https?://video1\.carambatv\.ru/v/)(?P<id>\d+)'
|
||||
_TESTS = [{
|
||||
'url': 'http://video1.carambatv.ru/v/191910501',
|
||||
'md5': '2f4a81b7cfd5ab866ee2d7270cb34a2a',
|
||||
'info_dict': {
|
||||
'id': '191910501',
|
||||
'ext': 'mp4',
|
||||
'title': '[BadComedian] - Разборка в Маниле (Абсолютный обзор)',
|
||||
'thumbnail': 're:^https?://.*\.jpg',
|
||||
'duration': 2678.31,
|
||||
},
|
||||
}, {
|
||||
'url': 'carambatv:191910501',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
|
||||
video = self._download_json(
|
||||
'http://video1.carambatv.ru/v/%s/videoinfo.js' % video_id,
|
||||
video_id)
|
||||
|
||||
title = video['title']
|
||||
|
||||
base_url = video.get('video') or 'http://video1.carambatv.ru/v/%s/' % video_id
|
||||
|
||||
formats = [{
|
||||
'url': base_url + f['fn'],
|
||||
'height': int_or_none(f.get('height')),
|
||||
'format_id': '%sp' % f['height'] if f.get('height') else None,
|
||||
} for f in video['qualities'] if f.get('fn')]
|
||||
self._sort_formats(formats)
|
||||
|
||||
thumbnail = video.get('splash')
|
||||
duration = float_or_none(try_get(
|
||||
video, lambda x: x['annotations'][0]['end_time'], compat_str))
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'title': title,
|
||||
'thumbnail': thumbnail,
|
||||
'duration': duration,
|
||||
'formats': formats,
|
||||
}
|
||||
|
||||
|
||||
class CarambaTVPageIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://carambatv\.ru/(?:[^/]+/)+(?P<id>[^/?#&]+)'
|
||||
_TEST = {
|
||||
'url': 'http://carambatv.ru/movie/bad-comedian/razborka-v-manile/',
|
||||
'md5': '',
|
||||
'info_dict': {
|
||||
'id': '191910501',
|
||||
'ext': 'mp4',
|
||||
'title': '[BadComedian] - Разборка в Маниле (Абсолютный обзор)',
|
||||
'thumbnail': 're:^https?://.*\.jpg$',
|
||||
'duration': 2678.31,
|
||||
},
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
video_url = self._og_search_property('video:iframe', webpage, default=None)
|
||||
|
||||
if not video_url:
|
||||
video_id = self._search_regex(
|
||||
r'(?:video_id|crmb_vuid)\s*[:=]\s*["\']?(\d+)',
|
||||
webpage, 'video id')
|
||||
video_url = 'carambatv:%s' % video_id
|
||||
|
||||
return self.url_result(video_url, CarambaTVIE.ie_key())
|
||||
@@ -58,8 +58,7 @@ class CDAIE(InfoExtractor):
|
||||
def extract_format(page, version):
|
||||
unpacked = decode_packed_codes(page)
|
||||
format_url = self._search_regex(
|
||||
r"(?:file|url)\s*:\s*(\\?[\"'])(?P<url>http.+?)\1", unpacked,
|
||||
'%s url' % version, fatal=False, group='url')
|
||||
r"url:\\'(.+?)\\'", unpacked, '%s url' % version, fatal=False)
|
||||
if not format_url:
|
||||
return
|
||||
f = {
|
||||
@@ -76,8 +75,7 @@ class CDAIE(InfoExtractor):
|
||||
info_dict['formats'].append(f)
|
||||
if not info_dict['duration']:
|
||||
info_dict['duration'] = parse_duration(self._search_regex(
|
||||
r"duration\s*:\s*(\\?[\"'])(?P<duration>.+?)\1",
|
||||
unpacked, 'duration', fatal=False, group='duration'))
|
||||
r"duration:\\'(.+?)\\'", unpacked, 'duration', fatal=False))
|
||||
|
||||
extract_format(webpage, 'default')
|
||||
|
||||
|
||||
@@ -20,64 +20,54 @@ class Channel9IE(InfoExtractor):
|
||||
'''
|
||||
IE_DESC = 'Channel 9'
|
||||
IE_NAME = 'channel9'
|
||||
_VALID_URL = r'https?://(?:www\.)?channel9\.msdn\.com/(?P<contentpath>.+?)(?P<rss>/RSS)?/?(?:[?#&]|$)'
|
||||
_VALID_URL = r'https?://(?:www\.)?channel9\.msdn\.com/(?P<contentpath>.+)/?'
|
||||
|
||||
_TESTS = [{
|
||||
'url': 'http://channel9.msdn.com/Events/TechEd/Australia/2013/KOS002',
|
||||
'md5': 'bbd75296ba47916b754e73c3a4bbdf10',
|
||||
'info_dict': {
|
||||
'id': 'Events/TechEd/Australia/2013/KOS002',
|
||||
'ext': 'mp4',
|
||||
'title': 'Developer Kick-Off Session: Stuff We Love',
|
||||
'description': 'md5:c08d72240b7c87fcecafe2692f80e35f',
|
||||
'duration': 4576,
|
||||
'thumbnail': 're:http://.*\.jpg',
|
||||
'session_code': 'KOS002',
|
||||
'session_day': 'Day 1',
|
||||
'session_room': 'Arena 1A',
|
||||
'session_speakers': ['Ed Blankenship', 'Andrew Coates', 'Brady Gaster', 'Patrick Klug',
|
||||
'Mads Kristensen'],
|
||||
_TESTS = [
|
||||
{
|
||||
'url': 'http://channel9.msdn.com/Events/TechEd/Australia/2013/KOS002',
|
||||
'md5': 'bbd75296ba47916b754e73c3a4bbdf10',
|
||||
'info_dict': {
|
||||
'id': 'Events/TechEd/Australia/2013/KOS002',
|
||||
'ext': 'mp4',
|
||||
'title': 'Developer Kick-Off Session: Stuff We Love',
|
||||
'description': 'md5:c08d72240b7c87fcecafe2692f80e35f',
|
||||
'duration': 4576,
|
||||
'thumbnail': 're:http://.*\.jpg',
|
||||
'session_code': 'KOS002',
|
||||
'session_day': 'Day 1',
|
||||
'session_room': 'Arena 1A',
|
||||
'session_speakers': ['Ed Blankenship', 'Andrew Coates', 'Brady Gaster', 'Patrick Klug', 'Mads Kristensen'],
|
||||
},
|
||||
},
|
||||
}, {
|
||||
'url': 'http://channel9.msdn.com/posts/Self-service-BI-with-Power-BI-nuclear-testing',
|
||||
'md5': 'b43ee4529d111bc37ba7ee4f34813e68',
|
||||
'info_dict': {
|
||||
'id': 'posts/Self-service-BI-with-Power-BI-nuclear-testing',
|
||||
'ext': 'mp4',
|
||||
'title': 'Self-service BI with Power BI - nuclear testing',
|
||||
'description': 'md5:d1e6ecaafa7fb52a2cacdf9599829f5b',
|
||||
'duration': 1540,
|
||||
'thumbnail': 're:http://.*\.jpg',
|
||||
'authors': ['Mike Wilmot'],
|
||||
{
|
||||
'url': 'http://channel9.msdn.com/posts/Self-service-BI-with-Power-BI-nuclear-testing',
|
||||
'md5': 'b43ee4529d111bc37ba7ee4f34813e68',
|
||||
'info_dict': {
|
||||
'id': 'posts/Self-service-BI-with-Power-BI-nuclear-testing',
|
||||
'ext': 'mp4',
|
||||
'title': 'Self-service BI with Power BI - nuclear testing',
|
||||
'description': 'md5:d1e6ecaafa7fb52a2cacdf9599829f5b',
|
||||
'duration': 1540,
|
||||
'thumbnail': 're:http://.*\.jpg',
|
||||
'authors': ['Mike Wilmot'],
|
||||
},
|
||||
},
|
||||
}, {
|
||||
# low quality mp4 is best
|
||||
'url': 'https://channel9.msdn.com/Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
|
||||
'info_dict': {
|
||||
'id': 'Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
|
||||
'ext': 'mp4',
|
||||
'title': 'Ranges for the Standard Library',
|
||||
'description': 'md5:2e6b4917677af3728c5f6d63784c4c5d',
|
||||
'duration': 5646,
|
||||
'thumbnail': 're:http://.*\.jpg',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
'url': 'https://channel9.msdn.com/Niners/Splendid22/Queue/76acff796e8f411184b008028e0d492b/RSS',
|
||||
'info_dict': {
|
||||
'id': 'Niners/Splendid22/Queue/76acff796e8f411184b008028e0d492b',
|
||||
'title': 'Channel 9',
|
||||
},
|
||||
'playlist_count': 2,
|
||||
}, {
|
||||
'url': 'https://channel9.msdn.com/Events/DEVintersection/DEVintersection-2016/RSS',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://channel9.msdn.com/Events/Speakers/scott-hanselman/RSS?UrlSafeName=scott-hanselman',
|
||||
'only_matching': True,
|
||||
}]
|
||||
{
|
||||
# low quality mp4 is best
|
||||
'url': 'https://channel9.msdn.com/Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
|
||||
'info_dict': {
|
||||
'id': 'Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
|
||||
'ext': 'mp4',
|
||||
'title': 'Ranges for the Standard Library',
|
||||
'description': 'md5:2e6b4917677af3728c5f6d63784c4c5d',
|
||||
'duration': 5646,
|
||||
'thumbnail': 're:http://.*\.jpg',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
_RSS_URL = 'http://channel9.msdn.com/%s/RSS'
|
||||
|
||||
@@ -264,30 +254,22 @@ class Channel9IE(InfoExtractor):
|
||||
|
||||
return self.playlist_result(contents)
|
||||
|
||||
def _extract_list(self, video_id, rss_url=None):
|
||||
if not rss_url:
|
||||
rss_url = self._RSS_URL % video_id
|
||||
rss = self._download_xml(rss_url, video_id, 'Downloading RSS')
|
||||
def _extract_list(self, content_path):
|
||||
rss = self._download_xml(self._RSS_URL % content_path, content_path, 'Downloading RSS')
|
||||
entries = [self.url_result(session_url.text, 'Channel9')
|
||||
for session_url in rss.findall('./channel/item/link')]
|
||||
title_text = rss.find('./channel/title').text
|
||||
return self.playlist_result(entries, video_id, title_text)
|
||||
return self.playlist_result(entries, content_path, title_text)
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
content_path = mobj.group('contentpath')
|
||||
rss = mobj.group('rss')
|
||||
|
||||
if rss:
|
||||
return self._extract_list(content_path, url)
|
||||
webpage = self._download_webpage(url, content_path, 'Downloading web page')
|
||||
|
||||
webpage = self._download_webpage(
|
||||
url, content_path, 'Downloading web page')
|
||||
|
||||
page_type = self._search_regex(
|
||||
r'<meta[^>]+name=(["\'])WT\.entryid\1[^>]+content=(["\'])(?P<pagetype>[^:]+).+?\2',
|
||||
webpage, 'page type', default=None, group='pagetype')
|
||||
if page_type:
|
||||
page_type_m = re.search(r'<meta name="WT.entryid" content="(?P<pagetype>[^:]+)[^"]+"/>', webpage)
|
||||
if page_type_m is not None:
|
||||
page_type = page_type_m.group('pagetype')
|
||||
if page_type == 'Entry': # Any 'item'-like page, may contain downloadable content
|
||||
return self._extract_entry_item(webpage, content_path)
|
||||
elif page_type == 'Session': # Event session page, may contain downloadable content
|
||||
@@ -296,5 +278,6 @@ class Channel9IE(InfoExtractor):
|
||||
return self._extract_list(content_path)
|
||||
else:
|
||||
raise ExtractorError('Unexpected WT.entryid %s' % page_type, expected=True)
|
||||
|
||||
else: # Assuming list
|
||||
return self._extract_list(content_path)
|
||||
|
||||
@@ -1,92 +0,0 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
|
||||
|
||||
class CloserToTruthIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?closertotruth\.com/(?:[^/]+/)*(?P<id>[^/?#&]+)'
|
||||
_TESTS = [{
|
||||
'url': 'http://closertotruth.com/series/solutions-the-mind-body-problem#video-3688',
|
||||
'info_dict': {
|
||||
'id': '0_zof1ktre',
|
||||
'display_id': 'solutions-the-mind-body-problem',
|
||||
'ext': 'mov',
|
||||
'title': 'Solutions to the Mind-Body Problem?',
|
||||
'upload_date': '20140221',
|
||||
'timestamp': 1392956007,
|
||||
'uploader_id': 'CTTXML'
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
'url': 'http://closertotruth.com/episodes/how-do-brains-work',
|
||||
'info_dict': {
|
||||
'id': '0_iuxai6g6',
|
||||
'display_id': 'how-do-brains-work',
|
||||
'ext': 'mov',
|
||||
'title': 'How do Brains Work?',
|
||||
'upload_date': '20140221',
|
||||
'timestamp': 1392956024,
|
||||
'uploader_id': 'CTTXML'
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
'url': 'http://closertotruth.com/interviews/1725',
|
||||
'info_dict': {
|
||||
'id': '1725',
|
||||
'title': 'AyaFr-002',
|
||||
},
|
||||
'playlist_mincount': 2,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
display_id = self._match_id(url)
|
||||
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
|
||||
partner_id = self._search_regex(
|
||||
r'<script[^>]+src=["\'].*?\b(?:partner_id|p)/(\d+)',
|
||||
webpage, 'kaltura partner_id')
|
||||
|
||||
title = self._search_regex(
|
||||
r'<title>(.+?)\s*\|\s*.+?</title>', webpage, 'video title')
|
||||
|
||||
select = self._search_regex(
|
||||
r'(?s)<select[^>]+id="select-version"[^>]*>(.+?)</select>',
|
||||
webpage, 'select version', default=None)
|
||||
if select:
|
||||
entry_ids = set()
|
||||
entries = []
|
||||
for mobj in re.finditer(
|
||||
r'<option[^>]+value=(["\'])(?P<id>[0-9a-z_]+)(?:#.+?)?\1[^>]*>(?P<title>[^<]+)',
|
||||
webpage):
|
||||
entry_id = mobj.group('id')
|
||||
if entry_id in entry_ids:
|
||||
continue
|
||||
entry_ids.add(entry_id)
|
||||
entries.append({
|
||||
'_type': 'url_transparent',
|
||||
'url': 'kaltura:%s:%s' % (partner_id, entry_id),
|
||||
'ie_key': 'Kaltura',
|
||||
'title': mobj.group('title'),
|
||||
})
|
||||
if entries:
|
||||
return self.playlist_result(entries, display_id, title)
|
||||
|
||||
entry_id = self._search_regex(
|
||||
r'<a[^>]+id=(["\'])embed-kaltura\1[^>]+data-kaltura=(["\'])(?P<id>[0-9a-z_]+)\2',
|
||||
webpage, 'kaltura entry_id', group='id')
|
||||
|
||||
return {
|
||||
'_type': 'url_transparent',
|
||||
'display_id': display_id,
|
||||
'url': 'kaltura:%s:%s' % (partner_id, entry_id),
|
||||
'ie_key': 'Kaltura',
|
||||
'title': title
|
||||
}
|
||||
@@ -45,7 +45,6 @@ from ..utils import (
|
||||
unescapeHTML,
|
||||
unified_strdate,
|
||||
url_basename,
|
||||
xpath_element,
|
||||
xpath_text,
|
||||
xpath_with_ns,
|
||||
determine_protocol,
|
||||
@@ -1031,7 +1030,7 @@ class InfoExtractor(object):
|
||||
if base_url:
|
||||
base_url = base_url.strip()
|
||||
|
||||
bootstrap_info = xpath_element(
|
||||
bootstrap_info = xpath_text(
|
||||
manifest, ['{http://ns.adobe.com/f4m/1.0}bootstrapInfo', '{http://ns.adobe.com/f4m/2.0}bootstrapInfo'],
|
||||
'bootstrap info', default=None)
|
||||
|
||||
@@ -1086,7 +1085,7 @@ class InfoExtractor(object):
|
||||
formats.append({
|
||||
'format_id': format_id,
|
||||
'url': manifest_url,
|
||||
'ext': 'flv' if bootstrap_info is not None else None,
|
||||
'ext': 'flv' if bootstrap_info else None,
|
||||
'tbr': tbr,
|
||||
'width': width,
|
||||
'height': height,
|
||||
|
||||
@@ -35,7 +35,6 @@ class DWIE(InfoExtractor):
|
||||
'upload_date': '20160311',
|
||||
}
|
||||
}, {
|
||||
# DW documentaries, only last for one or two weeks
|
||||
'url': 'http://www.dw.com/en/documentaries-welcome-to-the-90s-2016-05-21/e-19220158-9798',
|
||||
'md5': '56b6214ef463bfb9a3b71aeb886f3cf1',
|
||||
'info_dict': {
|
||||
@@ -45,7 +44,6 @@ class DWIE(InfoExtractor):
|
||||
'description': 'Welcome to the 90s - The Golden Decade of Hip Hop',
|
||||
'upload_date': '20160521',
|
||||
},
|
||||
'skip': 'Video removed',
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
|
||||
@@ -21,7 +21,6 @@ from .adobetv import (
|
||||
)
|
||||
from .adultswim import AdultSwimIE
|
||||
from .aenetworks import AENetworksIE
|
||||
from .afreecatv import AfreecaTVIE
|
||||
from .aftonbladet import AftonbladetIE
|
||||
from .airmozilla import AirMozillaIE
|
||||
from .aljazeera import AlJazeeraIE
|
||||
@@ -44,6 +43,7 @@ from .archiveorg import ArchiveOrgIE
|
||||
from .ard import (
|
||||
ARDIE,
|
||||
ARDMediathekIE,
|
||||
SportschauIE,
|
||||
)
|
||||
from .arte import (
|
||||
ArteTvIE,
|
||||
@@ -70,8 +70,6 @@ from .bandcamp import BandcampIE, BandcampAlbumIE
|
||||
from .bbc import (
|
||||
BBCCoUkIE,
|
||||
BBCCoUkArticleIE,
|
||||
BBCCoUkIPlayerPlaylistIE,
|
||||
BBCCoUkPlaylistIE,
|
||||
BBCIE,
|
||||
)
|
||||
from .beeg import BeegIE
|
||||
@@ -109,10 +107,6 @@ from .camwithher import CamWithHerIE
|
||||
from .canalplus import CanalplusIE
|
||||
from .canalc2 import Canalc2IE
|
||||
from .canvas import CanvasIE
|
||||
from .carambatv import (
|
||||
CarambaTVIE,
|
||||
CarambaTVPageIE,
|
||||
)
|
||||
from .cbc import (
|
||||
CBCIE,
|
||||
CBCPlayerIE,
|
||||
@@ -140,7 +134,6 @@ from .cliprs import ClipRsIE
|
||||
from .clipfish import ClipfishIE
|
||||
from .cliphunter import CliphunterIE
|
||||
from .clipsyndicate import ClipsyndicateIE
|
||||
from .closertotruth import CloserToTruthIE
|
||||
from .cloudy import CloudyIE
|
||||
from .clubic import ClubicIE
|
||||
from .clyp import ClypIE
|
||||
@@ -298,7 +291,6 @@ from .globo import (
|
||||
GloboArticleIE,
|
||||
)
|
||||
from .godtube import GodTubeIE
|
||||
from .godtv import GodTVIE
|
||||
from .goldenmoustache import GoldenMoustacheIE
|
||||
from .golem import GolemIE
|
||||
from .googledrive import GoogleDriveIE
|
||||
@@ -518,10 +510,7 @@ from .nhl import (
|
||||
NHLVideocenterCategoryIE,
|
||||
NHLIE,
|
||||
)
|
||||
from .nick import (
|
||||
NickIE,
|
||||
NickDeIE,
|
||||
)
|
||||
from .nick import NickIE
|
||||
from .niconico import NiconicoIE, NiconicoPlaylistIE
|
||||
from .ninegag import NineGagIE
|
||||
from .noco import NocoIE
|
||||
@@ -631,10 +620,7 @@ from .qqmusic import (
|
||||
QQMusicToplistIE,
|
||||
QQMusicPlaylistIE,
|
||||
)
|
||||
from .r7 import (
|
||||
R7IE,
|
||||
R7ArticleIE,
|
||||
)
|
||||
from .r7 import R7IE
|
||||
from .radiocanada import (
|
||||
RadioCanadaIE,
|
||||
RadioCanadaAudioVideoIE,
|
||||
@@ -661,7 +647,6 @@ from .revision3 import (
|
||||
from .rice import RICEIE
|
||||
from .ringtv import RingTVIE
|
||||
from .ro220 import Ro220IE
|
||||
from .rockstargames import RockstarGamesIE
|
||||
from .rottentomatoes import RottenTomatoesIE
|
||||
from .roxwel import RoxwelIE
|
||||
from .rtbf import RTBFIE
|
||||
@@ -750,7 +735,6 @@ from .sportbox import (
|
||||
SportBoxEmbedIE,
|
||||
)
|
||||
from .sportdeutschland import SportDeutschlandIE
|
||||
from .sportschau import SportschauIE
|
||||
from .srgssr import (
|
||||
SRGSSRIE,
|
||||
SRGSSRPlayIE,
|
||||
@@ -791,7 +775,6 @@ from .telecinco import TelecincoIE
|
||||
from .telegraaf import TelegraafIE
|
||||
from .telemb import TeleMBIE
|
||||
from .teletask import TeleTaskIE
|
||||
from .telewebion import TelewebionIE
|
||||
from .testurl import TestURLIE
|
||||
from .tf1 import TF1IE
|
||||
from .theintercept import TheInterceptIE
|
||||
@@ -876,7 +859,6 @@ from .twitch import (
|
||||
TwitchProfileIE,
|
||||
TwitchPastBroadcastsIE,
|
||||
TwitchStreamIE,
|
||||
TwitchClipsIE,
|
||||
)
|
||||
from .twitter import (
|
||||
TwitterCardIE,
|
||||
@@ -928,7 +910,6 @@ from .videomore import (
|
||||
)
|
||||
from .videopremium import VideoPremiumIE
|
||||
from .videott import VideoTtIE
|
||||
from .vidio import VidioIE
|
||||
from .vidme import (
|
||||
VidmeIE,
|
||||
VidmeUserIE,
|
||||
@@ -974,6 +955,7 @@ from .vporn import VpornIE
|
||||
from .vrt import VRTIE
|
||||
from .vube import VubeIE
|
||||
from .vuclip import VuClipIE
|
||||
from .vulture import VultureIE
|
||||
from .walla import WallaIE
|
||||
from .washingtonpost import (
|
||||
WashingtonPostIE,
|
||||
@@ -984,19 +966,18 @@ from .watchindianporn import WatchIndianPornIE
|
||||
from .wdr import (
|
||||
WDRIE,
|
||||
WDRMobileIE,
|
||||
WDRMausIE,
|
||||
)
|
||||
from .webofstories import (
|
||||
WebOfStoriesIE,
|
||||
WebOfStoriesPlaylistIE,
|
||||
)
|
||||
from .weibo import WeiboIE
|
||||
from .weiqitv import WeiqiTVIE
|
||||
from .wimp import WimpIE
|
||||
from .wistia import WistiaIE
|
||||
from .worldstarhiphop import WorldStarHipHopIE
|
||||
from .wrzuta import (
|
||||
WrzutaIE,
|
||||
WrzutaPlaylistIE,
|
||||
)
|
||||
from .wrzuta import WrzutaIE
|
||||
from .wsj import WSJIE
|
||||
from .xbef import XBefIE
|
||||
from .xboxclips import XboxClipsIE
|
||||
@@ -1032,10 +1013,7 @@ from .yesjapan import YesJapanIE
|
||||
from .yinyuetai import YinYueTaiIE
|
||||
from .ynet import YnetIE
|
||||
from .youjizz import YouJizzIE
|
||||
from .youku import (
|
||||
YoukuIE,
|
||||
YoukuShowIE,
|
||||
)
|
||||
from .youku import YoukuIE
|
||||
from .youporn import YouPornIE
|
||||
from .yourupload import YourUploadIE
|
||||
from .youtube import (
|
||||
|
||||
@@ -63,7 +63,6 @@ from .instagram import InstagramIE
|
||||
from .liveleak import LiveLeakIE
|
||||
from .threeqsdn import ThreeQSDNIE
|
||||
from .theplatform import ThePlatformIE
|
||||
from .vessel import VesselIE
|
||||
|
||||
|
||||
class GenericIE(InfoExtractor):
|
||||
@@ -627,13 +626,13 @@ class GenericIE(InfoExtractor):
|
||||
},
|
||||
# MTVSercices embed
|
||||
{
|
||||
'url': 'http://www.vulture.com/2016/06/new-key-peele-sketches-released.html',
|
||||
'md5': 'ca1aef97695ef2c1d6973256a57e5252',
|
||||
'url': 'http://www.gametrailers.com/news-post/76093/north-america-europe-is-getting-that-mario-kart-8-mercedes-dlc-too',
|
||||
'md5': '35727f82f58c76d996fc188f9755b0d5',
|
||||
'info_dict': {
|
||||
'id': '769f7ec0-0692-4d62-9b45-0d88074bffc1',
|
||||
'id': '0306a69b-8adf-4fb5-aace-75f8e8cbfca9',
|
||||
'ext': 'mp4',
|
||||
'title': 'Key and Peele|October 10, 2012|2|203|Liam Neesons - Uncensored',
|
||||
'description': 'Two valets share their love for movie star Liam Neesons.',
|
||||
'title': 'Review',
|
||||
'description': 'Mario\'s life in the fast lane has never looked so good.',
|
||||
},
|
||||
},
|
||||
# YouTube embed via <data-embed-url="">
|
||||
@@ -1032,17 +1031,6 @@ class GenericIE(InfoExtractor):
|
||||
'timestamp': 1389118457,
|
||||
},
|
||||
},
|
||||
# NBC News embed
|
||||
{
|
||||
'url': 'http://www.vulture.com/2016/06/letterman-couldnt-care-less-about-late-night.html',
|
||||
'md5': '1aa589c675898ae6d37a17913cf68d66',
|
||||
'info_dict': {
|
||||
'id': '701714499682',
|
||||
'ext': 'mp4',
|
||||
'title': 'PREVIEW: On Assignment: David Letterman',
|
||||
'description': 'A preview of Tom Brokaw\'s interview with David Letterman as part of the On Assignment series powered by Dateline. Airs Sunday June 12 at 7/6c.',
|
||||
},
|
||||
},
|
||||
# UDN embed
|
||||
{
|
||||
'url': 'https://video.udn.com/news/300346',
|
||||
@@ -1073,6 +1061,20 @@ class GenericIE(InfoExtractor):
|
||||
'skip_download': True,
|
||||
}
|
||||
},
|
||||
# Contains a SMIL manifest
|
||||
{
|
||||
'url': 'http://www.telewebion.com/fa/1263668/%D9%82%D8%B1%D8%B9%D9%87%E2%80%8C%DA%A9%D8%B4%DB%8C-%D9%84%DB%8C%DA%AF-%D9%82%D9%87%D8%B1%D9%85%D8%A7%D9%86%D8%A7%D9%86-%D8%A7%D8%B1%D9%88%D9%BE%D8%A7/%2B-%D9%81%D9%88%D8%AA%D8%A8%D8%A7%D9%84.html',
|
||||
'info_dict': {
|
||||
'id': 'file',
|
||||
'ext': 'flv',
|
||||
'title': '+ Football: Lottery Champions League Europe',
|
||||
'uploader': 'www.telewebion.com',
|
||||
},
|
||||
'params': {
|
||||
# rtmpe downloads
|
||||
'skip_download': True,
|
||||
}
|
||||
},
|
||||
# Brightcove URL in single quotes
|
||||
{
|
||||
'url': 'http://www.sportsnet.ca/baseball/mlb/sn-presents-russell-martin-world-citizen/',
|
||||
@@ -1531,11 +1533,6 @@ class GenericIE(InfoExtractor):
|
||||
if tp_urls:
|
||||
return _playlist_from_matches(tp_urls, ie='ThePlatform')
|
||||
|
||||
# Look for Vessel embeds
|
||||
vessel_urls = VesselIE._extract_urls(webpage)
|
||||
if vessel_urls:
|
||||
return _playlist_from_matches(vessel_urls, ie=VesselIE.ie_key())
|
||||
|
||||
# Look for embedded rtl.nl player
|
||||
matches = re.findall(
|
||||
r'<iframe[^>]+?src="((?:https?:)?//(?:www\.)?rtl\.nl/system/videoplayer/[^"]+(?:video_)?embed[^"]+)"',
|
||||
@@ -1843,6 +1840,14 @@ class GenericIE(InfoExtractor):
|
||||
url = unescapeHTML(mobj.group('url'))
|
||||
return self.url_result(url)
|
||||
|
||||
# Look for embedded vulture.com player
|
||||
mobj = re.search(
|
||||
r'<iframe src="(?P<url>https?://video\.vulture\.com/[^"]+)"',
|
||||
webpage)
|
||||
if mobj is not None:
|
||||
url = unescapeHTML(mobj.group('url'))
|
||||
return self.url_result(url, ie='Vulture')
|
||||
|
||||
# Look for embedded mtvservices player
|
||||
mtvservices_url = MTVServicesEmbeddedIE._extract_url(webpage)
|
||||
if mtvservices_url:
|
||||
@@ -1955,12 +1960,6 @@ class GenericIE(InfoExtractor):
|
||||
if nbc_sports_url:
|
||||
return self.url_result(nbc_sports_url, 'NBCSportsVPlayer')
|
||||
|
||||
# Look for NBC News embeds
|
||||
nbc_news_embed_url = re.search(
|
||||
r'<iframe[^>]+src=(["\'])(?P<url>(?:https?:)?//www\.nbcnews\.com/widget/video-embed/[^"\']+)\1', webpage)
|
||||
if nbc_news_embed_url:
|
||||
return self.url_result(nbc_news_embed_url.group('url'), 'NBCNews')
|
||||
|
||||
# Look for Google Drive embeds
|
||||
google_drive_url = GoogleDriveIE._extract_url(webpage)
|
||||
if google_drive_url:
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
from .common import InfoExtractor
|
||||
from .ooyala import OoyalaIE
|
||||
from ..utils import js_to_json
|
||||
|
||||
|
||||
class GodTVIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?god\.tv(?:/[^/]+)*/(?P<id>[^/?#&]+)'
|
||||
_TESTS = [{
|
||||
'url': 'http://god.tv/jesus-image/video/jesus-conference-2016/randy-needham',
|
||||
'info_dict': {
|
||||
'id': 'lpd3g2MzE6D1g8zFAKz8AGpxWcpu6o_3',
|
||||
'ext': 'mp4',
|
||||
'title': 'Randy Needham',
|
||||
'duration': 3615.08,
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
}
|
||||
}, {
|
||||
'url': 'http://god.tv/playlist/bible-study',
|
||||
'info_dict': {
|
||||
'id': 'bible-study',
|
||||
},
|
||||
'playlist_mincount': 37,
|
||||
}, {
|
||||
'url': 'http://god.tv/node/15097',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://god.tv/live/africa',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://god.tv/liveevents',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
display_id = self._match_id(url)
|
||||
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
|
||||
settings = self._parse_json(
|
||||
self._search_regex(
|
||||
r'jQuery\.extend\(Drupal\.settings\s*,\s*({.+?})\);',
|
||||
webpage, 'settings', default='{}'),
|
||||
display_id, transform_source=js_to_json, fatal=False)
|
||||
|
||||
ooyala_id = None
|
||||
|
||||
if settings:
|
||||
playlist = settings.get('playlist')
|
||||
if playlist and isinstance(playlist, list):
|
||||
entries = [
|
||||
OoyalaIE._build_url_result(video['content_id'])
|
||||
for video in playlist if video.get('content_id')]
|
||||
if entries:
|
||||
return self.playlist_result(entries, display_id)
|
||||
ooyala_id = settings.get('ooyala', {}).get('content_id')
|
||||
|
||||
if not ooyala_id:
|
||||
ooyala_id = self._search_regex(
|
||||
r'["\']content_id["\']\s*:\s*(["\'])(?P<id>[\w-]+)\1',
|
||||
webpage, 'ooyala id', group='id')
|
||||
|
||||
return OoyalaIE._build_url_result(ooyala_id)
|
||||
@@ -12,7 +12,7 @@ from ..utils import (
|
||||
class ImdbIE(InfoExtractor):
|
||||
IE_NAME = 'imdb'
|
||||
IE_DESC = 'Internet Movie Database trailers'
|
||||
_VALID_URL = r'https?://(?:www|m)\.imdb\.com/(?:video/[^/]+/|title/tt\d+.*?#lb-)vi(?P<id>\d+)'
|
||||
_VALID_URL = r'https?://(?:www|m)\.imdb\.com/video/[^/]+/vi(?P<id>\d+)'
|
||||
|
||||
_TESTS = [{
|
||||
'url': 'http://www.imdb.com/video/imdb/vi2524815897',
|
||||
@@ -25,12 +25,6 @@ class ImdbIE(InfoExtractor):
|
||||
}, {
|
||||
'url': 'http://www.imdb.com/video/_/vi2524815897',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://www.imdb.com/title/tt1667889/?ref_=ext_shr_eml_vi#lb-vi2524815897',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://www.imdb.com/title/tt1667889/#lb-vi2524815897',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
|
||||
@@ -60,8 +60,7 @@ class IndavideoEmbedIE(InfoExtractor):
|
||||
|
||||
formats = [{
|
||||
'url': video_url,
|
||||
'height': int_or_none(self._search_regex(
|
||||
r'\.(\d{3,4})\.mp4(?:\?|$)', video_url, 'height', default=None)),
|
||||
'height': self._search_regex(r'\.(\d{3,4})\.mp4$', video_url, 'height', default=None),
|
||||
} for video_url in video_urls]
|
||||
self._sort_formats(formats)
|
||||
|
||||
|
||||
@@ -8,7 +8,6 @@ from ..utils import (
|
||||
int_or_none,
|
||||
limit_length,
|
||||
lowercase_escape,
|
||||
try_get,
|
||||
)
|
||||
|
||||
|
||||
@@ -20,16 +19,10 @@ class InstagramIE(InfoExtractor):
|
||||
'info_dict': {
|
||||
'id': 'aye83DjauH',
|
||||
'ext': 'mp4',
|
||||
'uploader_id': 'naomipq',
|
||||
'title': 'Video by naomipq',
|
||||
'description': 'md5:1f17f0ab29bd6fe2bfad705f58de3cb8',
|
||||
'thumbnail': 're:^https?://.*\.jpg',
|
||||
'timestamp': 1371748545,
|
||||
'upload_date': '20130620',
|
||||
'uploader_id': 'naomipq',
|
||||
'uploader': 'Naomi Leonor Phan-Quang',
|
||||
'like_count': int,
|
||||
'comment_count': int,
|
||||
},
|
||||
}
|
||||
}, {
|
||||
# missing description
|
||||
'url': 'https://www.instagram.com/p/BA-pQFBG8HZ/?taken-by=britneyspears',
|
||||
@@ -38,13 +31,6 @@ class InstagramIE(InfoExtractor):
|
||||
'ext': 'mp4',
|
||||
'uploader_id': 'britneyspears',
|
||||
'title': 'Video by britneyspears',
|
||||
'thumbnail': 're:^https?://.*\.jpg',
|
||||
'timestamp': 1453760977,
|
||||
'upload_date': '20160125',
|
||||
'uploader_id': 'britneyspears',
|
||||
'uploader': 'Britney Spears',
|
||||
'like_count': int,
|
||||
'comment_count': int,
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
@@ -81,57 +67,21 @@ class InstagramIE(InfoExtractor):
|
||||
url = mobj.group('url')
|
||||
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
(video_url, description, thumbnail, timestamp, uploader,
|
||||
uploader_id, like_count, comment_count) = [None] * 8
|
||||
|
||||
shared_data = self._parse_json(
|
||||
self._search_regex(
|
||||
r'window\._sharedData\s*=\s*({.+?});',
|
||||
webpage, 'shared data', default='{}'),
|
||||
video_id, fatal=False)
|
||||
if shared_data:
|
||||
media = try_get(
|
||||
shared_data, lambda x: x['entry_data']['PostPage'][0]['media'], dict)
|
||||
if media:
|
||||
video_url = media.get('video_url')
|
||||
description = media.get('caption')
|
||||
thumbnail = media.get('display_src')
|
||||
timestamp = int_or_none(media.get('date'))
|
||||
uploader = media.get('owner', {}).get('full_name')
|
||||
uploader_id = media.get('owner', {}).get('username')
|
||||
like_count = int_or_none(media.get('likes', {}).get('count'))
|
||||
comment_count = int_or_none(media.get('comments', {}).get('count'))
|
||||
|
||||
if not video_url:
|
||||
video_url = self._og_search_video_url(webpage, secure=False)
|
||||
|
||||
if not uploader_id:
|
||||
uploader_id = self._search_regex(
|
||||
r'"owner"\s*:\s*{\s*"username"\s*:\s*"(.+?)"',
|
||||
webpage, 'uploader id', fatal=False)
|
||||
|
||||
if not description:
|
||||
description = self._search_regex(
|
||||
r'"caption"\s*:\s*"(.+?)"', webpage, 'description', default=None)
|
||||
if description is not None:
|
||||
description = lowercase_escape(description)
|
||||
|
||||
if not thumbnail:
|
||||
thumbnail = self._og_search_thumbnail(webpage)
|
||||
uploader_id = self._search_regex(r'"owner":{"username":"(.+?)"',
|
||||
webpage, 'uploader id', fatal=False)
|
||||
desc = self._search_regex(
|
||||
r'"caption":"(.+?)"', webpage, 'description', default=None)
|
||||
if desc is not None:
|
||||
desc = lowercase_escape(desc)
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'url': video_url,
|
||||
'url': self._og_search_video_url(webpage, secure=False),
|
||||
'ext': 'mp4',
|
||||
'title': 'Video by %s' % uploader_id,
|
||||
'description': description,
|
||||
'thumbnail': thumbnail,
|
||||
'timestamp': timestamp,
|
||||
'thumbnail': self._og_search_thumbnail(webpage),
|
||||
'uploader_id': uploader_id,
|
||||
'uploader': uploader,
|
||||
'like_count': like_count,
|
||||
'comment_count': comment_count,
|
||||
'description': desc,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -12,35 +12,9 @@ from ..utils import (
|
||||
|
||||
|
||||
class JWPlatformBaseIE(InfoExtractor):
|
||||
@staticmethod
|
||||
def _find_jwplayer_data(webpage):
|
||||
# TODO: Merge this with JWPlayer-related codes in generic.py
|
||||
|
||||
mobj = re.search(
|
||||
'jwplayer\((?P<quote>[\'"])[^\'" ]+(?P=quote)\)\.setup\((?P<options>[^)]+)\)',
|
||||
webpage)
|
||||
if mobj:
|
||||
return mobj.group('options')
|
||||
|
||||
def _extract_jwplayer_data(self, webpage, video_id, *args, **kwargs):
|
||||
jwplayer_data = self._parse_json(
|
||||
self._find_jwplayer_data(webpage), video_id)
|
||||
return self._parse_jwplayer_data(
|
||||
jwplayer_data, video_id, *args, **kwargs)
|
||||
|
||||
def _parse_jwplayer_data(self, jwplayer_data, video_id, require_title=True, m3u8_id=None, rtmp_params=None):
|
||||
# JWPlayer backward compatibility: flattened playlists
|
||||
# https://github.com/jwplayer/jwplayer/blob/v7.4.3/src/js/api/config.js#L81-L96
|
||||
if 'playlist' not in jwplayer_data:
|
||||
jwplayer_data = {'playlist': [jwplayer_data]}
|
||||
|
||||
video_data = jwplayer_data['playlist'][0]
|
||||
|
||||
# JWPlayer backward compatibility: flattened sources
|
||||
# https://github.com/jwplayer/jwplayer/blob/v7.4.3/src/js/playlist/item.js#L29-L35
|
||||
if 'sources' not in video_data:
|
||||
video_data['sources'] = [video_data]
|
||||
|
||||
formats = []
|
||||
for source in video_data['sources']:
|
||||
source_url = self._proto_relative_url(source['file'])
|
||||
|
||||
@@ -148,8 +148,8 @@ class KuwoAlbumIE(InfoExtractor):
|
||||
'url': 'http://www.kuwo.cn/album/502294/',
|
||||
'info_dict': {
|
||||
'id': '502294',
|
||||
'title': 'Made\xa0Series\xa0《M》',
|
||||
'description': 'md5:d463f0d8a0ff3c3ea3d6ed7452a9483f',
|
||||
'title': 'M',
|
||||
'description': 'md5:6a7235a84cc6400ec3b38a7bdaf1d60c',
|
||||
},
|
||||
'playlist_count': 2,
|
||||
}
|
||||
@@ -209,7 +209,7 @@ class KuwoSingerIE(InfoExtractor):
|
||||
'url': 'http://www.kuwo.cn/mingxing/bruno+mars/',
|
||||
'info_dict': {
|
||||
'id': 'bruno+mars',
|
||||
'title': 'Bruno\xa0Mars',
|
||||
'title': 'Bruno Mars',
|
||||
},
|
||||
'playlist_mincount': 329,
|
||||
}, {
|
||||
@@ -306,7 +306,7 @@ class KuwoMvIE(KuwoBaseIE):
|
||||
'id': '6480076',
|
||||
'ext': 'mp4',
|
||||
'title': 'My HouseMV',
|
||||
'creator': 'PM02:00',
|
||||
'creator': '2PM',
|
||||
},
|
||||
# In this video, music URLs (anti.s) are blocked outside China and
|
||||
# USA, while the MV URL (mvurl) is available globally, so force the MV
|
||||
|
||||
@@ -28,7 +28,7 @@ from ..utils import (
|
||||
|
||||
class LeIE(InfoExtractor):
|
||||
IE_DESC = '乐视网'
|
||||
_VALID_URL = r'https?://(?:www\.le\.com/ptv/vplay|sports\.le\.com/video)/(?P<id>\d+)\.html'
|
||||
_VALID_URL = r'https?://www\.le\.com/ptv/vplay/(?P<id>\d+)\.html'
|
||||
|
||||
_URL_TEMPLATE = 'http://www.le.com/ptv/vplay/%s.html'
|
||||
|
||||
@@ -69,9 +69,6 @@ class LeIE(InfoExtractor):
|
||||
'hls_prefer_native': True,
|
||||
},
|
||||
'skip': 'Only available in China',
|
||||
}, {
|
||||
'url': 'http://sports.le.com/video/25737697.html',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
@staticmethod
|
||||
@@ -199,7 +196,7 @@ class LeIE(InfoExtractor):
|
||||
|
||||
|
||||
class LePlaylistIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://[a-z]+\.le\.com/(?!video)[a-z]+/(?P<id>[a-z0-9_]+)'
|
||||
_VALID_URL = r'https?://[a-z]+\.le\.com/[a-z]+/(?P<id>[a-z0-9_]+)'
|
||||
|
||||
_TESTS = [{
|
||||
'url': 'http://www.le.com/tv/46177.html',
|
||||
|
||||
@@ -1,24 +1,20 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
|
||||
from ..utils import (
|
||||
determine_ext,
|
||||
float_or_none,
|
||||
int_or_none,
|
||||
parse_filesize,
|
||||
)
|
||||
|
||||
|
||||
class LibraryOfCongressIE(InfoExtractor):
|
||||
IE_NAME = 'loc'
|
||||
IE_DESC = 'Library of Congress'
|
||||
_VALID_URL = r'https?://(?:www\.)?loc\.gov/(?:item/|today/cyberlc/feature_wdesc\.php\?.*\brec=)(?P<id>[0-9]+)'
|
||||
_TESTS = [{
|
||||
# embedded via <div class="media-player"
|
||||
_VALID_URL = r'https?://(?:www\.)?loc\.gov/item/(?P<id>[0-9]+)'
|
||||
_TEST = {
|
||||
'url': 'http://loc.gov/item/90716351/',
|
||||
'md5': '353917ff7f0255aa6d4b80a034833de8',
|
||||
'info_dict': {
|
||||
@@ -29,35 +25,7 @@ class LibraryOfCongressIE(InfoExtractor):
|
||||
'duration': 0,
|
||||
'view_count': int,
|
||||
},
|
||||
}, {
|
||||
# webcast embedded via mediaObjectId
|
||||
'url': 'https://www.loc.gov/today/cyberlc/feature_wdesc.php?rec=5578',
|
||||
'info_dict': {
|
||||
'id': '5578',
|
||||
'ext': 'mp4',
|
||||
'title': 'Help! Preservation Training Needs Here, There & Everywhere',
|
||||
'duration': 3765,
|
||||
'view_count': int,
|
||||
'subtitles': 'mincount:1',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
# with direct download links
|
||||
'url': 'https://www.loc.gov/item/78710669/',
|
||||
'info_dict': {
|
||||
'id': '78710669',
|
||||
'ext': 'mp4',
|
||||
'title': 'La vie et la passion de Jesus-Christ',
|
||||
'duration': 0,
|
||||
'view_count': int,
|
||||
'formats': 'mincount:4',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}]
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
@@ -66,20 +34,18 @@ class LibraryOfCongressIE(InfoExtractor):
|
||||
media_id = self._search_regex(
|
||||
(r'id=(["\'])media-player-(?P<id>.+?)\1',
|
||||
r'<video[^>]+id=(["\'])uuid-(?P<id>.+?)\1',
|
||||
r'<video[^>]+data-uuid=(["\'])(?P<id>.+?)\1',
|
||||
r'mediaObjectId\s*:\s*(["\'])(?P<id>.+?)\1'),
|
||||
r'<video[^>]+data-uuid=(["\'])(?P<id>.+?)\1'),
|
||||
webpage, 'media id', group='id')
|
||||
|
||||
data = self._download_json(
|
||||
'https://media.loc.gov/services/v1/media?id=%s&context=json' % media_id,
|
||||
data = self._parse_json(
|
||||
self._download_webpage(
|
||||
'https://media.loc.gov/services/v1/media?id=%s&context=json' % media_id,
|
||||
video_id),
|
||||
video_id)['mediaObject']
|
||||
|
||||
derivative = data['derivatives'][0]
|
||||
media_url = derivative['derivativeUrl']
|
||||
|
||||
title = derivative.get('shortName') or data.get('shortName') or self._og_search_title(
|
||||
webpage)
|
||||
|
||||
# Following algorithm was extracted from setAVSource js function
|
||||
# found in webpage
|
||||
media_url = media_url.replace('rtmp', 'https')
|
||||
@@ -95,7 +61,6 @@ class LibraryOfCongressIE(InfoExtractor):
|
||||
'format_id': 'hls',
|
||||
'ext': 'mp4',
|
||||
'protocol': 'm3u8_native',
|
||||
'quality': 1,
|
||||
}]
|
||||
elif 'vod/mp3:' in media_url:
|
||||
formats = [{
|
||||
@@ -103,41 +68,17 @@ class LibraryOfCongressIE(InfoExtractor):
|
||||
'vcodec': 'none',
|
||||
}]
|
||||
|
||||
download_urls = set()
|
||||
for m in re.finditer(
|
||||
r'<option[^>]+value=(["\'])(?P<url>.+?)\1[^>]+data-file-download=[^>]+>\s*(?P<id>.+?)(?:(?: |\s+)\((?P<size>.+?)\))?\s*<', webpage):
|
||||
format_id = m.group('id').lower()
|
||||
if format_id == 'gif':
|
||||
continue
|
||||
download_url = m.group('url')
|
||||
if download_url in download_urls:
|
||||
continue
|
||||
download_urls.add(download_url)
|
||||
formats.append({
|
||||
'url': download_url,
|
||||
'format_id': format_id,
|
||||
'filesize_approx': parse_filesize(m.group('size')),
|
||||
})
|
||||
|
||||
self._sort_formats(formats)
|
||||
|
||||
title = derivative.get('shortName') or data.get('shortName') or self._og_search_title(webpage)
|
||||
duration = float_or_none(data.get('duration'))
|
||||
view_count = int_or_none(data.get('viewCount'))
|
||||
|
||||
subtitles = {}
|
||||
cc_url = data.get('ccUrl')
|
||||
if cc_url:
|
||||
subtitles.setdefault('en', []).append({
|
||||
'url': cc_url,
|
||||
'ext': 'ttml',
|
||||
})
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'title': title,
|
||||
'thumbnail': self._og_search_thumbnail(webpage, default=None),
|
||||
'thumbnail': self._og_search_thumbnail(webpage),
|
||||
'duration': duration,
|
||||
'view_count': view_count,
|
||||
'formats': formats,
|
||||
'subtitles': subtitles,
|
||||
}
|
||||
|
||||
@@ -98,19 +98,13 @@ class LimelightBaseIE(InfoExtractor):
|
||||
} for thumbnail in properties.get('thumbnails', []) if thumbnail.get('url')]
|
||||
|
||||
subtitles = {}
|
||||
for caption in properties.get('captions', []):
|
||||
for caption in properties.get('captions', {}):
|
||||
lang = caption.get('language_code')
|
||||
subtitles_url = caption.get('url')
|
||||
if lang and subtitles_url:
|
||||
subtitles.setdefault(lang, []).append({
|
||||
subtitles[lang] = [{
|
||||
'url': subtitles_url,
|
||||
})
|
||||
closed_captions_url = properties.get('closed_captions_url')
|
||||
if closed_captions_url:
|
||||
subtitles.setdefault('en', []).append({
|
||||
'url': closed_captions_url,
|
||||
'ext': 'ttml',
|
||||
})
|
||||
}]
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
@@ -129,18 +123,7 @@ class LimelightBaseIE(InfoExtractor):
|
||||
|
||||
class LimelightMediaIE(LimelightBaseIE):
|
||||
IE_NAME = 'limelight'
|
||||
_VALID_URL = r'''(?x)
|
||||
(?:
|
||||
limelight:media:|
|
||||
https?://
|
||||
(?:
|
||||
link\.videoplatform\.limelight\.com/media/|
|
||||
assets\.delvenetworks\.com/player/loader\.swf
|
||||
)
|
||||
\?.*?\bmediaId=
|
||||
)
|
||||
(?P<id>[a-z0-9]{32})
|
||||
'''
|
||||
_VALID_URL = r'(?:limelight:media:|https?://link\.videoplatform\.limelight\.com/media/\??\bmediaId=)(?P<id>[a-z0-9]{32})'
|
||||
_TESTS = [{
|
||||
'url': 'http://link.videoplatform.limelight.com/media/?mediaId=3ffd040b522b4485b6d84effc750cd86',
|
||||
'info_dict': {
|
||||
@@ -175,9 +158,6 @@ class LimelightMediaIE(LimelightBaseIE):
|
||||
# rtmp download
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
'url': 'https://assets.delvenetworks.com/player/loader.swf?mediaId=8018a574f08d416e95ceaccae4ba0452',
|
||||
'only_matching': True,
|
||||
}]
|
||||
_PLAYLIST_SERVICE_PATH = 'media'
|
||||
_API_PATH = 'media'
|
||||
@@ -196,29 +176,15 @@ class LimelightMediaIE(LimelightBaseIE):
|
||||
|
||||
class LimelightChannelIE(LimelightBaseIE):
|
||||
IE_NAME = 'limelight:channel'
|
||||
_VALID_URL = r'''(?x)
|
||||
(?:
|
||||
limelight:channel:|
|
||||
https?://
|
||||
(?:
|
||||
link\.videoplatform\.limelight\.com/media/|
|
||||
assets\.delvenetworks\.com/player/loader\.swf
|
||||
)
|
||||
\?.*?\bchannelId=
|
||||
)
|
||||
(?P<id>[a-z0-9]{32})
|
||||
'''
|
||||
_TESTS = [{
|
||||
_VALID_URL = r'(?:limelight:channel:|https?://link\.videoplatform\.limelight\.com/media/\??\bchannelId=)(?P<id>[a-z0-9]{32})'
|
||||
_TEST = {
|
||||
'url': 'http://link.videoplatform.limelight.com/media/?channelId=ab6a524c379342f9b23642917020c082',
|
||||
'info_dict': {
|
||||
'id': 'ab6a524c379342f9b23642917020c082',
|
||||
'title': 'Javascript Sample Code',
|
||||
},
|
||||
'playlist_mincount': 3,
|
||||
}, {
|
||||
'url': 'http://assets.delvenetworks.com/player/loader.swf?channelId=ab6a524c379342f9b23642917020c082',
|
||||
'only_matching': True,
|
||||
}]
|
||||
}
|
||||
_PLAYLIST_SERVICE_PATH = 'channel'
|
||||
_API_PATH = 'channels'
|
||||
|
||||
@@ -241,29 +207,15 @@ class LimelightChannelIE(LimelightBaseIE):
|
||||
|
||||
class LimelightChannelListIE(LimelightBaseIE):
|
||||
IE_NAME = 'limelight:channel_list'
|
||||
_VALID_URL = r'''(?x)
|
||||
(?:
|
||||
limelight:channel_list:|
|
||||
https?://
|
||||
(?:
|
||||
link\.videoplatform\.limelight\.com/media/|
|
||||
assets\.delvenetworks\.com/player/loader\.swf
|
||||
)
|
||||
\?.*?\bchannelListId=
|
||||
)
|
||||
(?P<id>[a-z0-9]{32})
|
||||
'''
|
||||
_TESTS = [{
|
||||
_VALID_URL = r'(?:limelight:channel_list:|https?://link\.videoplatform\.limelight\.com/media/\?.*?\bchannelListId=)(?P<id>[a-z0-9]{32})'
|
||||
_TEST = {
|
||||
'url': 'http://link.videoplatform.limelight.com/media/?channelListId=301b117890c4465c8179ede21fd92e2b',
|
||||
'info_dict': {
|
||||
'id': '301b117890c4465c8179ede21fd92e2b',
|
||||
'title': 'Website - Hero Player',
|
||||
},
|
||||
'playlist_mincount': 2,
|
||||
}, {
|
||||
'url': 'https://assets.delvenetworks.com/player/loader.swf?channelListId=301b117890c4465c8179ede21fd92e2b',
|
||||
'only_matching': True,
|
||||
}]
|
||||
}
|
||||
_PLAYLIST_SERVICE_PATH = 'channel_list'
|
||||
|
||||
def _real_extract(self, url):
|
||||
|
||||
@@ -203,10 +203,9 @@ class LivestreamIE(InfoExtractor):
|
||||
if not videos_info:
|
||||
break
|
||||
for v in videos_info:
|
||||
v_id = compat_str(v['id'])
|
||||
entries.append(self.url_result(
|
||||
'http://livestream.com/accounts/%s/events/%s/videos/%s' % (account_id, event_id, v_id),
|
||||
'Livestream', v_id, v.get('caption')))
|
||||
'http://livestream.com/accounts/%s/events/%s/videos/%s' % (account_id, event_id, v['id']),
|
||||
'Livestream', v['id'], v['caption']))
|
||||
last_video = videos_info[-1]['id']
|
||||
return self.playlist_result(entries, event_id, event_data['full_name'])
|
||||
|
||||
|
||||
@@ -1,100 +1,100 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import re
|
||||
import json
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..compat import (
|
||||
compat_HTTPError,
|
||||
compat_str,
|
||||
compat_urlparse,
|
||||
)
|
||||
from ..compat import compat_str
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
clean_html,
|
||||
int_or_none,
|
||||
sanitized_Request,
|
||||
urlencode_postdata,
|
||||
)
|
||||
|
||||
|
||||
class LyndaBaseIE(InfoExtractor):
|
||||
_SIGNIN_URL = 'https://www.lynda.com/signin'
|
||||
_PASSWORD_URL = 'https://www.lynda.com/signin/password'
|
||||
_USER_URL = 'https://www.lynda.com/signin/user'
|
||||
_LOGIN_URL = 'https://www.lynda.com/login/login.aspx'
|
||||
_ACCOUNT_CREDENTIALS_HINT = 'Use --username and --password options to provide lynda.com account credentials.'
|
||||
_NETRC_MACHINE = 'lynda'
|
||||
|
||||
def _real_initialize(self):
|
||||
self._login()
|
||||
|
||||
@staticmethod
|
||||
def _check_error(json_string, key_or_keys):
|
||||
keys = [key_or_keys] if isinstance(key_or_keys, compat_str) else key_or_keys
|
||||
for key in keys:
|
||||
error = json_string.get(key)
|
||||
if error:
|
||||
raise ExtractorError('Unable to login: %s' % error, expected=True)
|
||||
|
||||
def _login_step(self, form_html, fallback_action_url, extra_form_data, note, referrer_url):
|
||||
action_url = self._search_regex(
|
||||
r'<form[^>]+action=(["\'])(?P<url>.+?)\1', form_html,
|
||||
'post url', default=fallback_action_url, group='url')
|
||||
|
||||
if not action_url.startswith('http'):
|
||||
action_url = compat_urlparse.urljoin(self._SIGNIN_URL, action_url)
|
||||
|
||||
form_data = self._hidden_inputs(form_html)
|
||||
form_data.update(extra_form_data)
|
||||
|
||||
try:
|
||||
response = self._download_json(
|
||||
action_url, None, note,
|
||||
data=urlencode_postdata(form_data),
|
||||
headers={
|
||||
'Referer': referrer_url,
|
||||
'X-Requested-With': 'XMLHttpRequest',
|
||||
})
|
||||
except ExtractorError as e:
|
||||
if isinstance(e.cause, compat_HTTPError) and e.cause.code == 500:
|
||||
response = self._parse_json(e.cause.read().decode('utf-8'), None)
|
||||
self._check_error(response, ('email', 'password'))
|
||||
raise
|
||||
|
||||
self._check_error(response, 'ErrorMessage')
|
||||
|
||||
return response, action_url
|
||||
|
||||
def _login(self):
|
||||
username, password = self._get_login_info()
|
||||
if username is None:
|
||||
return
|
||||
|
||||
# Step 1: download signin page
|
||||
signin_page = self._download_webpage(
|
||||
self._SIGNIN_URL, None, 'Downloading signin page')
|
||||
login_form = {
|
||||
'username': username,
|
||||
'password': password,
|
||||
'remember': 'false',
|
||||
'stayPut': 'false'
|
||||
}
|
||||
request = sanitized_Request(
|
||||
self._LOGIN_URL, urlencode_postdata(login_form))
|
||||
login_page = self._download_webpage(
|
||||
request, None, 'Logging in as %s' % username)
|
||||
|
||||
# Already logged in
|
||||
if any(re.search(p, signin_page) for p in (
|
||||
'isLoggedIn\s*:\s*true', r'logout\.aspx', r'>Log out<')):
|
||||
# Not (yet) logged in
|
||||
m = re.search(r'loginResultJson\s*=\s*\'(?P<json>[^\']+)\';', login_page)
|
||||
if m is not None:
|
||||
response = m.group('json')
|
||||
response_json = json.loads(response)
|
||||
state = response_json['state']
|
||||
|
||||
if state == 'notlogged':
|
||||
raise ExtractorError(
|
||||
'Unable to login, incorrect username and/or password',
|
||||
expected=True)
|
||||
|
||||
# This is when we get popup:
|
||||
# > You're already logged in to lynda.com on two devices.
|
||||
# > If you log in here, we'll log you out of another device.
|
||||
# So, we need to confirm this.
|
||||
if state == 'conflicted':
|
||||
confirm_form = {
|
||||
'username': '',
|
||||
'password': '',
|
||||
'resolve': 'true',
|
||||
'remember': 'false',
|
||||
'stayPut': 'false',
|
||||
}
|
||||
request = sanitized_Request(
|
||||
self._LOGIN_URL, urlencode_postdata(confirm_form))
|
||||
login_page = self._download_webpage(
|
||||
request, None,
|
||||
'Confirming log in and log out from another device')
|
||||
|
||||
if all(not re.search(p, login_page) for p in ('isLoggedIn\s*:\s*true', r'logout\.aspx', r'>Log out<')):
|
||||
if 'login error' in login_page:
|
||||
mobj = re.search(
|
||||
r'(?s)<h1[^>]+class="topmost">(?P<title>[^<]+)</h1>\s*<div>(?P<description>.+?)</div>',
|
||||
login_page)
|
||||
if mobj:
|
||||
raise ExtractorError(
|
||||
'lynda returned error: %s - %s'
|
||||
% (mobj.group('title'), clean_html(mobj.group('description'))),
|
||||
expected=True)
|
||||
raise ExtractorError('Unable to log in')
|
||||
|
||||
def _logout(self):
|
||||
username, _ = self._get_login_info()
|
||||
if username is None:
|
||||
return
|
||||
|
||||
# Step 2: submit email
|
||||
signin_form = self._search_regex(
|
||||
r'(?s)(<form[^>]+data-form-name=["\']signin["\'][^>]*>.+?</form>)',
|
||||
signin_page, 'signin form')
|
||||
signin_page, signin_url = self._login_step(
|
||||
signin_form, self._PASSWORD_URL, {'email': username},
|
||||
'Submitting email', self._SIGNIN_URL)
|
||||
|
||||
# Step 3: submit password
|
||||
password_form = signin_page['body']
|
||||
self._login_step(
|
||||
password_form, self._USER_URL, {'email': username, 'password': password},
|
||||
'Submitting password', signin_url)
|
||||
self._download_webpage(
|
||||
'http://www.lynda.com/ajax/logout.aspx', None,
|
||||
'Logging out', 'Unable to log out', fatal=False)
|
||||
|
||||
|
||||
class LyndaIE(LyndaBaseIE):
|
||||
IE_NAME = 'lynda'
|
||||
IE_DESC = 'lynda.com videos'
|
||||
_VALID_URL = r'https?://www\.lynda\.com/(?:[^/]+/[^/]+/\d+|player/embed)/(?P<id>\d+)'
|
||||
_NETRC_MACHINE = 'lynda'
|
||||
|
||||
_TIMECODE_REGEX = r'\[(?P<timecode>\d+:\d+:\d+[\.,]\d+)\]'
|
||||
|
||||
@@ -212,6 +212,8 @@ class LyndaCourseIE(LyndaBaseIE):
|
||||
'http://www.lynda.com/ajax/player?courseId=%s&type=course' % course_id,
|
||||
course_id, 'Downloading course JSON')
|
||||
|
||||
self._logout()
|
||||
|
||||
if course.get('Status') == 'NotFound':
|
||||
raise ExtractorError(
|
||||
'Course %s does not exist' % course_id, expected=True)
|
||||
@@ -244,6 +246,5 @@ class LyndaCourseIE(LyndaBaseIE):
|
||||
% unaccessible_videos + self._ACCOUNT_CREDENTIALS_HINT)
|
||||
|
||||
course_title = course.get('Title')
|
||||
course_description = course.get('Description')
|
||||
|
||||
return self.playlist_result(entries, course_id, course_title, course_description)
|
||||
return self.playlist_result(entries, course_id, course_title)
|
||||
|
||||
@@ -4,12 +4,16 @@ from __future__ import unicode_literals
|
||||
import random
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import xpath_text
|
||||
from ..compat import compat_urllib_parse_urlencode
|
||||
from ..utils import (
|
||||
sanitized_Request,
|
||||
xpath_text,
|
||||
)
|
||||
|
||||
|
||||
class MatchTVIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://matchtv\.ru(?:/on-air|/?#live-player)'
|
||||
_TESTS = [{
|
||||
_VALID_URL = r'https?://matchtv\.ru/?#live-player'
|
||||
_TEST = {
|
||||
'url': 'http://matchtv.ru/#live-player',
|
||||
'info_dict': {
|
||||
'id': 'matchtv-live',
|
||||
@@ -20,16 +24,12 @@ class MatchTVIE(InfoExtractor):
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
'url': 'http://matchtv.ru/on-air/',
|
||||
'only_matching': True,
|
||||
}]
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = 'matchtv-live'
|
||||
video_url = self._download_json(
|
||||
'http://player.matchtv.ntvplus.tv/player/smil', video_id,
|
||||
query={
|
||||
request = sanitized_Request(
|
||||
'http://player.matchtv.ntvplus.tv/player/smil?%s' % compat_urllib_parse_urlencode({
|
||||
'ts': '',
|
||||
'quality': 'SD',
|
||||
'contentId': '561d2c0df7159b37178b4567',
|
||||
@@ -40,10 +40,11 @@ class MatchTVIE(InfoExtractor):
|
||||
'contentType': 'channel',
|
||||
'timeShift': '0',
|
||||
'platform': 'portal',
|
||||
},
|
||||
}),
|
||||
headers={
|
||||
'Referer': 'http://player.matchtv.ntvplus.tv/embed-player/NTVEmbedPlayer.swf',
|
||||
})['data']['videoUrl']
|
||||
})
|
||||
video_url = self._download_json(request, video_id)['data']['videoUrl']
|
||||
f4m_url = xpath_text(self._download_xml(video_url, video_id), './to')
|
||||
formats = self._extract_f4m_formats(f4m_url, video_id)
|
||||
self._sort_formats(formats)
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..compat import (
|
||||
compat_urllib_parse_urlencode,
|
||||
@@ -11,7 +8,6 @@ from ..compat import (
|
||||
from ..utils import (
|
||||
get_element_by_attribute,
|
||||
int_or_none,
|
||||
remove_start,
|
||||
)
|
||||
|
||||
|
||||
@@ -19,7 +15,7 @@ class MiTeleIE(InfoExtractor):
|
||||
IE_DESC = 'mitele.es'
|
||||
_VALID_URL = r'https?://www\.mitele\.es/[^/]+/[^/]+/[^/]+/(?P<id>[^/]+)/'
|
||||
|
||||
_TESTS = [{
|
||||
_TEST = {
|
||||
'url': 'http://www.mitele.es/programas-tv/diario-de/la-redaccion/programa-144/',
|
||||
# MD5 is unstable
|
||||
'info_dict': {
|
||||
@@ -28,31 +24,10 @@ class MiTeleIE(InfoExtractor):
|
||||
'ext': 'flv',
|
||||
'title': 'Tor, la web invisible',
|
||||
'description': 'md5:3b6fce7eaa41b2d97358726378d9369f',
|
||||
'series': 'Diario de',
|
||||
'season': 'La redacción',
|
||||
'episode': 'Programa 144',
|
||||
'thumbnail': 're:(?i)^https?://.*\.jpg$',
|
||||
'duration': 2913,
|
||||
},
|
||||
}, {
|
||||
# no explicit title
|
||||
'url': 'http://www.mitele.es/programas-tv/cuarto-milenio/temporada-6/programa-226/',
|
||||
'info_dict': {
|
||||
'id': 'eLZSwoEd1S3pVyUm8lc6F',
|
||||
'display_id': 'programa-226',
|
||||
'ext': 'flv',
|
||||
'title': 'Cuarto Milenio - Temporada 6 - Programa 226',
|
||||
'description': 'md5:50daf9fadefa4e62d9fc866d0c015701',
|
||||
'series': 'Cuarto Milenio',
|
||||
'season': 'Temporada 6',
|
||||
'episode': 'Programa 226',
|
||||
'thumbnail': 're:(?i)^https?://.*\.jpg$',
|
||||
'duration': 7312,
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}]
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
display_id = self._match_id(url)
|
||||
@@ -95,22 +70,7 @@ class MiTeleIE(InfoExtractor):
|
||||
self._sort_formats(formats)
|
||||
|
||||
title = self._search_regex(
|
||||
r'class="Destacado-text"[^>]*>\s*<strong>([^<]+)</strong>',
|
||||
webpage, 'title', default=None)
|
||||
|
||||
mobj = re.search(r'''(?sx)
|
||||
class="Destacado-text"[^>]*>.*?<h1>\s*
|
||||
<span>(?P<series>[^<]+)</span>\s*
|
||||
<span>(?P<season>[^<]+)</span>\s*
|
||||
<span>(?P<episode>[^<]+)</span>''', webpage)
|
||||
series, season, episode = mobj.groups() if mobj else [None] * 3
|
||||
|
||||
if not title:
|
||||
if mobj:
|
||||
title = '%s - %s - %s' % (series, season, episode)
|
||||
else:
|
||||
title = remove_start(self._search_regex(
|
||||
r'<title>([^<]+)</title>', webpage, 'title'), 'Ver online ')
|
||||
r'class="Destacado-text"[^>]*>\s*<strong>([^<]+)</strong>', webpage, 'title')
|
||||
|
||||
video_id = self._search_regex(
|
||||
r'data-media-id\s*=\s*"([^"]+)"', webpage,
|
||||
@@ -123,9 +83,6 @@ class MiTeleIE(InfoExtractor):
|
||||
'display_id': display_id,
|
||||
'title': title,
|
||||
'description': get_element_by_attribute('class', 'text', webpage),
|
||||
'series': series,
|
||||
'season': season,
|
||||
'episode': episode,
|
||||
'thumbnail': thumbnail,
|
||||
'duration': duration,
|
||||
'formats': formats,
|
||||
|
||||
@@ -6,7 +6,6 @@ from .common import InfoExtractor
|
||||
from ..compat import (
|
||||
compat_urllib_parse_urlencode,
|
||||
compat_str,
|
||||
compat_xpath,
|
||||
)
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
@@ -85,10 +84,9 @@ class MTVServicesInfoExtractor(InfoExtractor):
|
||||
rtmp_video_url = rendition.find('./src').text
|
||||
if rtmp_video_url.endswith('siteunavail.png'):
|
||||
continue
|
||||
new_url = self._transform_rtmp_url(rtmp_video_url)
|
||||
formats.append({
|
||||
'ext': 'flv' if new_url.startswith('rtmp') else ext,
|
||||
'url': new_url,
|
||||
'ext': ext,
|
||||
'url': self._transform_rtmp_url(rtmp_video_url),
|
||||
'format_id': rendition.get('bitrate'),
|
||||
'width': int(rendition.get('width')),
|
||||
'height': int(rendition.get('height')),
|
||||
@@ -141,9 +139,9 @@ class MTVServicesInfoExtractor(InfoExtractor):
|
||||
itemdoc, './/{http://search.yahoo.com/mrss/}category',
|
||||
'scheme', 'urn:mtvn:video_title')
|
||||
if title_el is None:
|
||||
title_el = itemdoc.find(compat_xpath('.//{http://search.yahoo.com/mrss/}title'))
|
||||
title_el = itemdoc.find('.//{http://search.yahoo.com/mrss/}title')
|
||||
if title_el is None:
|
||||
title_el = itemdoc.find(compat_xpath('.//title'))
|
||||
title_el = itemdoc.find('.//title') or itemdoc.find('./title')
|
||||
if title_el.text is None:
|
||||
title_el = None
|
||||
|
||||
|
||||
@@ -67,23 +67,6 @@ class NBCIE(InfoExtractor):
|
||||
# This video has expired but with an escaped embedURL
|
||||
'url': 'http://www.nbc.com/parenthood/episode-guide/season-5/just-like-at-home/515',
|
||||
'only_matching': True,
|
||||
},
|
||||
{
|
||||
# HLS streams requires the 'hdnea3' cookie
|
||||
'url': 'http://www.nbc.com/Kings/video/goliath/n1806',
|
||||
'info_dict': {
|
||||
'id': 'n1806',
|
||||
'ext': 'mp4',
|
||||
'title': 'Goliath',
|
||||
'description': 'When an unknown soldier saves the life of the King\'s son in battle, he\'s thrust into the limelight and politics of the kingdom.',
|
||||
'timestamp': 1237100400,
|
||||
'upload_date': '20090315',
|
||||
'uploader': 'NBCU-COM',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
'skip': 'Only works from US',
|
||||
}
|
||||
]
|
||||
|
||||
@@ -266,11 +249,6 @@ class NBCNewsIE(ThePlatformIE):
|
||||
'url': 'http://www.nbcnews.com/watch/dateline/full-episode--deadly-betrayal-386250819952',
|
||||
'only_matching': True,
|
||||
},
|
||||
{
|
||||
# From http://www.vulture.com/2016/06/letterman-couldnt-care-less-about-late-night.html
|
||||
'url': 'http://www.nbcnews.com/widget/video-embed/701714499682',
|
||||
'only_matching': True,
|
||||
},
|
||||
]
|
||||
|
||||
def _real_extract(self, url):
|
||||
@@ -294,17 +272,18 @@ class NBCNewsIE(ThePlatformIE):
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
info = None
|
||||
bootstrap_json = self._search_regex(
|
||||
[r'(?m)(?:var\s+(?:bootstrapJson|playlistData)|NEWS\.videoObj)\s*=\s*({.+});?\s*$',
|
||||
r'videoObj\s*:\s*({.+})', r'data-video="([^"]+)"'],
|
||||
r'(?m)var\s+(?:bootstrapJson|playlistData)\s*=\s*({.+});?\s*$',
|
||||
webpage, 'bootstrap json', default=None)
|
||||
bootstrap = self._parse_json(
|
||||
bootstrap_json, display_id, transform_source=unescapeHTML)
|
||||
if 'results' in bootstrap:
|
||||
if bootstrap_json:
|
||||
bootstrap = self._parse_json(bootstrap_json, display_id)
|
||||
info = bootstrap['results'][0]['video']
|
||||
elif 'video' in bootstrap:
|
||||
info = bootstrap['video']
|
||||
else:
|
||||
info = bootstrap
|
||||
player_instance_json = self._search_regex(
|
||||
r'videoObj\s*:\s*({.+})', webpage, 'player instance', default=None)
|
||||
if not player_instance_json:
|
||||
player_instance_json = self._html_search_regex(
|
||||
r'data-video="([^"]+)"', webpage, 'video json')
|
||||
info = self._parse_json(player_instance_json, display_id)
|
||||
video_id = info['mpxId']
|
||||
title = info['title']
|
||||
|
||||
|
||||
@@ -3,7 +3,6 @@ from __future__ import unicode_literals
|
||||
|
||||
from .mtv import MTVServicesInfoExtractor
|
||||
from ..compat import compat_urllib_parse_urlencode
|
||||
from ..utils import update_url_query
|
||||
|
||||
|
||||
class NickIE(MTVServicesInfoExtractor):
|
||||
@@ -62,26 +61,3 @@ class NickIE(MTVServicesInfoExtractor):
|
||||
|
||||
def _extract_mgid(self, webpage):
|
||||
return self._search_regex(r'data-contenturi="([^"]+)', webpage, 'mgid')
|
||||
|
||||
|
||||
class NickDeIE(MTVServicesInfoExtractor):
|
||||
IE_NAME = 'nick.de'
|
||||
_VALID_URL = r'https?://(?:www\.)?nick\.de/(?:playlist|shows)/(?:[^/]+/)*(?P<id>[^/?#&]+)'
|
||||
_TESTS = [{
|
||||
'url': 'http://www.nick.de/playlist/3773-top-videos/videos/episode/17306-zu-wasser-und-zu-land-rauchende-erdnusse',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://www.nick.de/shows/342-icarly',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
mrss_url = update_url_query(self._search_regex(
|
||||
r'data-mrss=(["\'])(?P<url>http.+?)\1', webpage, 'mrss url', group='url'),
|
||||
{'siteKey': 'nick.de'})
|
||||
|
||||
return self._get_videos_info_from_url(mrss_url, video_id)
|
||||
|
||||
@@ -163,7 +163,7 @@ class NRKTVIE(NRKBaseIE):
|
||||
'ext': 'mp4',
|
||||
'title': '20 spørsmål 23.05.2014',
|
||||
'description': 'md5:bdea103bc35494c143c6a9acdd84887a',
|
||||
'duration': 1741,
|
||||
'duration': 1741.52,
|
||||
},
|
||||
}, {
|
||||
'url': 'https://tv.nrk.no/program/mdfp15000514',
|
||||
@@ -173,7 +173,7 @@ class NRKTVIE(NRKBaseIE):
|
||||
'ext': 'mp4',
|
||||
'title': 'Grunnlovsjubiléet - Stor ståhei for ingenting 24.05.2014',
|
||||
'description': 'md5:89290c5ccde1b3a24bb8050ab67fe1db',
|
||||
'duration': 4605,
|
||||
'duration': 4605.08,
|
||||
},
|
||||
}, {
|
||||
# single playlist video
|
||||
@@ -260,34 +260,30 @@ class NRKPlaylistIE(InfoExtractor):
|
||||
|
||||
class NRKSkoleIE(InfoExtractor):
|
||||
IE_DESC = 'NRK Skole'
|
||||
_VALID_URL = r'https?://(?:www\.)?nrk\.no/skole/?\?.*\bmediaId=(?P<id>\d+)'
|
||||
_VALID_URL = r'https?://(?:www\.)?nrk\.no/skole/klippdetalj?.*\btopic=(?P<id>[^/?#&]+)'
|
||||
|
||||
_TESTS = [{
|
||||
'url': 'https://www.nrk.no/skole/?page=search&q=&mediaId=14099',
|
||||
'md5': '6bc936b01f9dd8ed45bc58b252b2d9b6',
|
||||
'url': 'http://nrk.no/skole/klippdetalj?topic=nrk:klipp/616532',
|
||||
'md5': '04cd85877cc1913bce73c5d28a47e00f',
|
||||
'info_dict': {
|
||||
'id': '6021',
|
||||
'ext': 'mp4',
|
||||
'ext': 'flv',
|
||||
'title': 'Genetikk og eneggede tvillinger',
|
||||
'description': 'md5:3aca25dcf38ec30f0363428d2b265f8d',
|
||||
'duration': 399,
|
||||
},
|
||||
}, {
|
||||
'url': 'https://www.nrk.no/skole/?page=objectives&subject=naturfag&objective=K15114&mediaId=19355',
|
||||
'url': 'http://www.nrk.no/skole/klippdetalj?topic=nrk%3Aklipp%2F616532#embed',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://www.nrk.no/skole/klippdetalj?topic=urn:x-mediadb:21379',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
video_id = compat_urllib_parse_unquote(self._match_id(url))
|
||||
|
||||
webpage = self._download_webpage(
|
||||
'https://mimir.nrk.no/plugin/1.0/static?mediaId=%s' % video_id,
|
||||
video_id)
|
||||
|
||||
nrk_id = self._parse_json(
|
||||
self._search_regex(
|
||||
r'<script[^>]+type=["\']application/json["\'][^>]*>({.+?})</script>',
|
||||
webpage, 'application json'),
|
||||
video_id)['activeMedia']['psId']
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
nrk_id = self._search_regex(r'data-nrk-id=["\'](\d+)', webpage, 'nrk id')
|
||||
return self.url_result('nrk:%s' % nrk_id)
|
||||
|
||||
@@ -14,7 +14,7 @@ from ..utils import (
|
||||
|
||||
|
||||
class OpenloadIE(InfoExtractor):
|
||||
_VALID_URL = r'https://openload.(?:co|io)/(?:f|embed)/(?P<id>[a-zA-Z0-9-_]+)'
|
||||
_VALID_URL = r'https://openload.(?:co|io)/(?:f|embed)/(?P<id>[a-zA-Z0-9-]+)'
|
||||
|
||||
_TESTS = [{
|
||||
'url': 'https://openload.co/f/kUEfGclsU9o',
|
||||
@@ -31,9 +31,6 @@ class OpenloadIE(InfoExtractor):
|
||||
}, {
|
||||
'url': 'https://openload.io/f/ZAn6oz-VZGE/',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://openload.co/f/_-ztPaZtMhM/',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
# unavailable via https://openload.co/f/Sxz5sADo82g/, different layout
|
||||
# for title and ext
|
||||
@@ -103,7 +100,7 @@ class OpenloadIE(InfoExtractor):
|
||||
raise ExtractorError('File not found', expected=True)
|
||||
|
||||
code = self._search_regex(
|
||||
r'</video>\s*</div>\s*<script[^>]+>[^>]+</script>\s*<script[^>]+>([^<]+)</script>',
|
||||
r'</video>\s*</div>\s*<script[^>]+>([^<]+)</script>',
|
||||
webpage, 'JS code')
|
||||
|
||||
decoded = self.openload_decode(code)
|
||||
|
||||
@@ -1,32 +1,19 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import re
|
||||
import json
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
int_or_none,
|
||||
js_to_json,
|
||||
qualities,
|
||||
)
|
||||
|
||||
|
||||
class PornHdIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?pornhd\.com/(?:[a-z]{2,4}/)?videos/(?P<id>\d+)(?:/(?P<display_id>.+))?'
|
||||
_TESTS = [{
|
||||
'url': 'http://www.pornhd.com/videos/9864/selfie-restroom-masturbation-fun-with-chubby-cutie-hd-porn-video',
|
||||
'md5': 'c8b964b1f0a4b5f7f28ae3a5c9f86ad5',
|
||||
'info_dict': {
|
||||
'id': '9864',
|
||||
'display_id': 'selfie-restroom-masturbation-fun-with-chubby-cutie-hd-porn-video',
|
||||
'ext': 'mp4',
|
||||
'title': 'Restroom selfie masturbation',
|
||||
'description': 'md5:3748420395e03e31ac96857a8f125b2b',
|
||||
'thumbnail': 're:^https?://.*\.jpg',
|
||||
'view_count': int,
|
||||
'age_limit': 18,
|
||||
}
|
||||
}, {
|
||||
# removed video
|
||||
_TEST = {
|
||||
'url': 'http://www.pornhd.com/videos/1962/sierra-day-gets-his-cum-all-over-herself-hd-porn-video',
|
||||
'md5': '956b8ca569f7f4d8ec563e2c41598441',
|
||||
'info_dict': {
|
||||
@@ -38,9 +25,8 @@ class PornHdIE(InfoExtractor):
|
||||
'thumbnail': 're:^https?://.*\.jpg',
|
||||
'view_count': int,
|
||||
'age_limit': 18,
|
||||
},
|
||||
'skip': 'Not available anymore',
|
||||
}]
|
||||
}
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
@@ -52,38 +38,28 @@ class PornHdIE(InfoExtractor):
|
||||
title = self._html_search_regex(
|
||||
[r'<span[^>]+class=["\']video-name["\'][^>]*>([^<]+)',
|
||||
r'<title>(.+?) - .*?[Pp]ornHD.*?</title>'], webpage, 'title')
|
||||
|
||||
sources = self._parse_json(js_to_json(self._search_regex(
|
||||
r"(?s)'sources'\s*:\s*(\{.+?\})\s*\}[;,)]",
|
||||
webpage, 'sources', default='{}')), video_id)
|
||||
|
||||
if not sources:
|
||||
message = self._html_search_regex(
|
||||
r'(?s)<(div|p)[^>]+class="no-video"[^>]*>(?P<value>.+?)</\1',
|
||||
webpage, 'error message', group='value')
|
||||
raise ExtractorError('%s said: %s' % (self.IE_NAME, message), expected=True)
|
||||
|
||||
formats = []
|
||||
for format_id, video_url in sources.items():
|
||||
if not video_url:
|
||||
continue
|
||||
height = int_or_none(self._search_regex(
|
||||
r'^(\d+)[pP]', format_id, 'height', default=None))
|
||||
formats.append({
|
||||
'url': video_url,
|
||||
'format_id': format_id,
|
||||
'height': height,
|
||||
})
|
||||
self._sort_formats(formats)
|
||||
|
||||
description = self._html_search_regex(
|
||||
r'<(div|p)[^>]+class="description"[^>]*>(?P<value>[^<]+)</\1',
|
||||
webpage, 'description', fatal=False, group='value')
|
||||
r'<div class="description">([^<]+)</div>', webpage, 'description', fatal=False)
|
||||
view_count = int_or_none(self._html_search_regex(
|
||||
r'(\d+) views\s*<', webpage, 'view count', fatal=False))
|
||||
r'(\d+) views\s*</span>', webpage, 'view count', fatal=False))
|
||||
thumbnail = self._search_regex(
|
||||
r"'poster'\s*:\s*'([^']+)'", webpage, 'thumbnail', fatal=False)
|
||||
|
||||
quality = qualities(['sd', 'hd'])
|
||||
sources = json.loads(js_to_json(self._search_regex(
|
||||
r"(?s)'sources'\s*:\s*(\{.+?\})\s*\}[;,)]",
|
||||
webpage, 'sources')))
|
||||
formats = []
|
||||
for qname, video_url in sources.items():
|
||||
if not video_url:
|
||||
continue
|
||||
formats.append({
|
||||
'url': video_url,
|
||||
'format_id': qname,
|
||||
'quality': quality(qname),
|
||||
})
|
||||
self._sort_formats(formats)
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'display_id': display_id,
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import itertools
|
||||
@@ -40,25 +39,7 @@ class PornHubIE(InfoExtractor):
|
||||
'dislike_count': int,
|
||||
'comment_count': int,
|
||||
'age_limit': 18,
|
||||
},
|
||||
}, {
|
||||
# non-ASCII title
|
||||
'url': 'http://www.pornhub.com/view_video.php?viewkey=1331683002',
|
||||
'info_dict': {
|
||||
'id': '1331683002',
|
||||
'ext': 'mp4',
|
||||
'title': '重庆婷婷女王足交',
|
||||
'uploader': 'cj397186295',
|
||||
'duration': 1753,
|
||||
'view_count': int,
|
||||
'like_count': int,
|
||||
'dislike_count': int,
|
||||
'comment_count': int,
|
||||
'age_limit': 18,
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}
|
||||
}, {
|
||||
'url': 'http://www.pornhub.com/view_video.php?viewkey=ph557bbb6676d2d',
|
||||
'only_matching': True,
|
||||
@@ -95,25 +76,19 @@ class PornHubIE(InfoExtractor):
|
||||
'PornHub said: %s' % error_msg,
|
||||
expected=True, video_id=video_id)
|
||||
|
||||
# video_title from flashvars contains whitespace instead of non-ASCII (see
|
||||
# http://www.pornhub.com/view_video.php?viewkey=1331683002), not relying
|
||||
# on that anymore.
|
||||
title = self._html_search_meta(
|
||||
'twitter:title', webpage, default=None) or self._search_regex(
|
||||
(r'<h1[^>]+class=["\']title["\'][^>]*>(?P<title>[^<]+)',
|
||||
r'<div[^>]+data-video-title=(["\'])(?P<title>.+?)\1',
|
||||
r'shareTitle\s*=\s*(["\'])(?P<title>.+?)\1'),
|
||||
webpage, 'title', group='title')
|
||||
|
||||
flashvars = self._parse_json(
|
||||
self._search_regex(
|
||||
r'var\s+flashvars_\d+\s*=\s*({.+?});', webpage, 'flashvars', default='{}'),
|
||||
video_id)
|
||||
if flashvars:
|
||||
video_title = flashvars.get('video_title')
|
||||
thumbnail = flashvars.get('image_url')
|
||||
duration = int_or_none(flashvars.get('video_duration'))
|
||||
else:
|
||||
title, thumbnail, duration = [None] * 3
|
||||
video_title, thumbnail, duration = [None] * 3
|
||||
|
||||
if not video_title:
|
||||
video_title = self._html_search_regex(r'<h1 [^>]+>([^<]+)', webpage, 'title')
|
||||
|
||||
video_uploader = self._html_search_regex(
|
||||
r'(?s)From: .+?<(?:a href="/users/|a href="/channels/|span class="username)[^>]+>(.+?)<',
|
||||
@@ -162,7 +137,7 @@ class PornHubIE(InfoExtractor):
|
||||
return {
|
||||
'id': video_id,
|
||||
'uploader': video_uploader,
|
||||
'title': title,
|
||||
'title': video_title,
|
||||
'thumbnail': thumbnail,
|
||||
'duration': duration,
|
||||
'view_count': view_count,
|
||||
|
||||
@@ -2,19 +2,22 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import int_or_none
|
||||
from ..utils import (
|
||||
js_to_json,
|
||||
unescapeHTML,
|
||||
int_or_none,
|
||||
)
|
||||
|
||||
|
||||
class R7IE(InfoExtractor):
|
||||
_VALID_URL = r'''(?x)
|
||||
https?://
|
||||
_VALID_URL = r'''(?x)https?://
|
||||
(?:
|
||||
(?:[a-zA-Z]+)\.r7\.com(?:/[^/]+)+/idmedia/|
|
||||
noticias\.r7\.com(?:/[^/]+)+/[^/]+-|
|
||||
player\.r7\.com/video/i/
|
||||
)
|
||||
(?P<id>[\da-f]{24})
|
||||
'''
|
||||
'''
|
||||
_TESTS = [{
|
||||
'url': 'http://videos.r7.com/policiais-humilham-suspeito-a-beira-da-morte-morre-com-dignidade-/idmedia/54e7050b0cf2ff57e0279389.html',
|
||||
'md5': '403c4e393617e8e8ddc748978ee8efde',
|
||||
@@ -22,7 +25,6 @@ class R7IE(InfoExtractor):
|
||||
'id': '54e7050b0cf2ff57e0279389',
|
||||
'ext': 'mp4',
|
||||
'title': 'Policiais humilham suspeito à beira da morte: "Morre com dignidade"',
|
||||
'description': 'md5:01812008664be76a6479aa58ec865b72',
|
||||
'thumbnail': 're:^https?://.*\.jpg$',
|
||||
'duration': 98,
|
||||
'like_count': int,
|
||||
@@ -42,72 +44,45 @@ class R7IE(InfoExtractor):
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
|
||||
video = self._download_json(
|
||||
'http://player-api.r7.com/video/i/%s' % video_id, video_id)
|
||||
webpage = self._download_webpage(
|
||||
'http://player.r7.com/video/i/%s' % video_id, video_id)
|
||||
|
||||
title = video['title']
|
||||
item = self._parse_json(js_to_json(self._search_regex(
|
||||
r'(?s)var\s+item\s*=\s*({.+?});', webpage, 'player')), video_id)
|
||||
|
||||
title = unescapeHTML(item['title'])
|
||||
thumbnail = item.get('init', {}).get('thumbUri')
|
||||
duration = None
|
||||
|
||||
statistics = item.get('statistics', {})
|
||||
like_count = int_or_none(statistics.get('likes'))
|
||||
view_count = int_or_none(statistics.get('views'))
|
||||
|
||||
formats = []
|
||||
media_url_hls = video.get('media_url_hls')
|
||||
if media_url_hls:
|
||||
formats.extend(self._extract_m3u8_formats(
|
||||
media_url_hls, video_id, 'mp4', entry_protocol='m3u8_native',
|
||||
m3u8_id='hls', fatal=False))
|
||||
media_url = video.get('media_url')
|
||||
if media_url:
|
||||
f = {
|
||||
'url': media_url,
|
||||
'format_id': 'http',
|
||||
}
|
||||
# m3u8 format always matches the http format, let's copy metadata from
|
||||
# one to another
|
||||
m3u8_formats = list(filter(
|
||||
lambda f: f.get('vcodec') != 'none' and f.get('resolution') != 'multiple',
|
||||
formats))
|
||||
if len(m3u8_formats) == 1:
|
||||
f_copy = m3u8_formats[0].copy()
|
||||
f_copy.update(f)
|
||||
f_copy['protocol'] = 'http'
|
||||
f = f_copy
|
||||
formats.append(f)
|
||||
for format_key, format_dict in item['playlist'][0].items():
|
||||
src = format_dict.get('src')
|
||||
if not src:
|
||||
continue
|
||||
format_id = format_dict.get('format') or format_key
|
||||
if duration is None:
|
||||
duration = format_dict.get('duration')
|
||||
if '.f4m' in src:
|
||||
formats.extend(self._extract_f4m_formats(src, video_id, preference=-1))
|
||||
elif src.endswith('.m3u8'):
|
||||
formats.extend(self._extract_m3u8_formats(src, video_id, 'mp4', preference=-2))
|
||||
else:
|
||||
formats.append({
|
||||
'url': src,
|
||||
'format_id': format_id,
|
||||
})
|
||||
self._sort_formats(formats)
|
||||
|
||||
description = video.get('description')
|
||||
thumbnail = video.get('thumb')
|
||||
duration = int_or_none(video.get('media_duration'))
|
||||
like_count = int_or_none(video.get('likes'))
|
||||
view_count = int_or_none(video.get('views'))
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'title': title,
|
||||
'description': description,
|
||||
'thumbnail': thumbnail,
|
||||
'duration': duration,
|
||||
'like_count': like_count,
|
||||
'view_count': view_count,
|
||||
'formats': formats,
|
||||
}
|
||||
|
||||
|
||||
class R7ArticleIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:[a-zA-Z]+)\.r7\.com/(?:[^/]+/)+[^/?#&]+-(?P<id>\d+)'
|
||||
_TEST = {
|
||||
'url': 'http://tv.r7.com/record-play/balanco-geral/videos/policiais-humilham-suspeito-a-beira-da-morte-morre-com-dignidade-16102015',
|
||||
'only_matching': True,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def suitable(cls, url):
|
||||
return False if R7IE.suitable(url) else super(R7ArticleIE, cls).suitable(url)
|
||||
|
||||
def _real_extract(self, url):
|
||||
display_id = self._match_id(url)
|
||||
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
|
||||
video_id = self._search_regex(
|
||||
r'<div[^>]+(?:id=["\']player-|class=["\']embed["\'][^>]+id=["\'])([\da-f]{24})',
|
||||
webpage, 'video id')
|
||||
|
||||
return self.url_result('http://player.r7.com/video/i/%s' % video_id, R7IE.ie_key())
|
||||
|
||||
@@ -1,69 +0,0 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
int_or_none,
|
||||
parse_iso8601,
|
||||
)
|
||||
|
||||
|
||||
class RockstarGamesIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?rockstargames\.com/videos(?:/video/|#?/?\?.*\bvideo=)(?P<id>\d+)'
|
||||
_TESTS = [{
|
||||
'url': 'https://www.rockstargames.com/videos/video/11544/',
|
||||
'md5': '03b5caa6e357a4bd50e3143fc03e5733',
|
||||
'info_dict': {
|
||||
'id': '11544',
|
||||
'ext': 'mp4',
|
||||
'title': 'Further Adventures in Finance and Felony Trailer',
|
||||
'description': 'md5:6d31f55f30cb101b5476c4a379e324a3',
|
||||
'thumbnail': 're:^https?://.*\.jpg$',
|
||||
'timestamp': 1464876000,
|
||||
'upload_date': '20160602',
|
||||
}
|
||||
}, {
|
||||
'url': 'http://www.rockstargames.com/videos#/?video=48',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
|
||||
video = self._download_json(
|
||||
'https://www.rockstargames.com/videoplayer/videos/get-video.json',
|
||||
video_id, query={
|
||||
'id': video_id,
|
||||
'locale': 'en_us',
|
||||
})['video']
|
||||
|
||||
title = video['title']
|
||||
|
||||
formats = []
|
||||
for video in video['files_processed']['video/mp4']:
|
||||
if not video.get('src'):
|
||||
continue
|
||||
resolution = video.get('resolution')
|
||||
height = int_or_none(self._search_regex(
|
||||
r'^(\d+)[pP]$', resolution or '', 'height', default=None))
|
||||
formats.append({
|
||||
'url': self._proto_relative_url(video['src']),
|
||||
'format_id': resolution,
|
||||
'height': height,
|
||||
})
|
||||
|
||||
if not formats:
|
||||
youtube_id = video.get('youtube_id')
|
||||
if youtube_id:
|
||||
return self.url_result(youtube_id, 'Youtube')
|
||||
|
||||
self._sort_formats(formats)
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'title': title,
|
||||
'description': video.get('description'),
|
||||
'thumbnail': self._proto_relative_url(video.get('screencap')),
|
||||
'timestamp': parse_iso8601(video.get('created')),
|
||||
'formats': formats,
|
||||
}
|
||||
@@ -1,38 +0,0 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
from .wdr import WDRBaseIE
|
||||
from ..utils import get_element_by_attribute
|
||||
|
||||
|
||||
class SportschauIE(WDRBaseIE):
|
||||
IE_NAME = 'Sportschau'
|
||||
_VALID_URL = r'https?://(?:www\.)?sportschau\.de/(?:[^/]+/)+video-?(?P<id>[^/#?]+)\.html'
|
||||
_TEST = {
|
||||
'url': 'http://www.sportschau.de/uefaeuro2016/videos/video-dfb-team-geht-gut-gelaunt-ins-spiel-gegen-polen-100.html',
|
||||
'info_dict': {
|
||||
'id': 'mdb-1140188',
|
||||
'display_id': 'dfb-team-geht-gut-gelaunt-ins-spiel-gegen-polen-100',
|
||||
'ext': 'mp4',
|
||||
'title': 'DFB-Team geht gut gelaunt ins Spiel gegen Polen',
|
||||
'description': 'Vor dem zweiten Gruppenspiel gegen Polen herrscht gute Stimmung im deutschen Team. Insbesondere Bastian Schweinsteiger strotzt vor Optimismus nach seinem Tor gegen die Ukraine.',
|
||||
'upload_date': '20160615',
|
||||
},
|
||||
'skip': 'Geo-restricted to Germany',
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
title = get_element_by_attribute('class', 'headline', webpage)
|
||||
description = self._html_search_meta('description', webpage, 'description')
|
||||
|
||||
info = self._extract_wdr_video(webpage, video_id)
|
||||
|
||||
info.update({
|
||||
'title': title,
|
||||
'description': description,
|
||||
})
|
||||
|
||||
return info
|
||||
@@ -5,7 +5,6 @@ import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
sanitized_Request,
|
||||
urlencode_postdata,
|
||||
)
|
||||
@@ -15,7 +14,7 @@ class StreamcloudIE(InfoExtractor):
|
||||
IE_NAME = 'streamcloud.eu'
|
||||
_VALID_URL = r'https?://streamcloud\.eu/(?P<id>[a-zA-Z0-9_-]+)(?:/(?P<fname>[^#?]*)\.html)?'
|
||||
|
||||
_TESTS = [{
|
||||
_TEST = {
|
||||
'url': 'http://streamcloud.eu/skp9j99s4bpz/youtube-dl_test_video_____________-BaW_jenozKc.mp4.html',
|
||||
'md5': '6bea4c7fa5daaacc2a946b7146286686',
|
||||
'info_dict': {
|
||||
@@ -24,10 +23,7 @@ class StreamcloudIE(InfoExtractor):
|
||||
'title': 'youtube-dl test video \'/\\ ä ↭',
|
||||
},
|
||||
'skip': 'Only available from the EU'
|
||||
}, {
|
||||
'url': 'http://streamcloud.eu/ua8cmfh1nbe6/NSHIP-148--KUC-NG--H264-.mp4.html',
|
||||
'only_matching': True,
|
||||
}]
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
@@ -35,10 +31,6 @@ class StreamcloudIE(InfoExtractor):
|
||||
|
||||
orig_webpage = self._download_webpage(url, video_id)
|
||||
|
||||
if '>File Not Found<' in orig_webpage:
|
||||
raise ExtractorError(
|
||||
'Video %s does not exist' % video_id, expected=True)
|
||||
|
||||
fields = re.findall(r'''(?x)<input\s+
|
||||
type="(?:hidden|submit)"\s+
|
||||
name="([^"]+)"\s+
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
from .common import InfoExtractor
|
||||
|
||||
|
||||
class TelewebionIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://www\.telewebion\.com/#!/episode/(?P<id>\d+)'
|
||||
|
||||
_TEST = {
|
||||
'url': 'http://www.telewebion.com/#!/episode/1263668/',
|
||||
'info_dict': {
|
||||
'id': '1263668',
|
||||
'ext': 'mp4',
|
||||
'title': 'قرعه\u200cکشی لیگ قهرمانان اروپا',
|
||||
'thumbnail': 're:^https?://.*\.jpg',
|
||||
'view_count': int,
|
||||
},
|
||||
'params': {
|
||||
# m3u8 download
|
||||
'skip_download': True,
|
||||
},
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
|
||||
secure_token = self._download_webpage(
|
||||
'http://m.s2.telewebion.com/op/op?action=getSecurityToken', video_id)
|
||||
episode_details = self._download_json(
|
||||
'http://m.s2.telewebion.com/op/op', video_id,
|
||||
query={'action': 'getEpisodeDetails', 'episode_id': video_id})
|
||||
|
||||
m3u8_url = 'http://m.s1.telewebion.com/smil/%s.m3u8?filepath=%s&m3u8=1&secure_token=%s' % (
|
||||
video_id, episode_details['file_path'], secure_token)
|
||||
formats = self._extract_m3u8_formats(
|
||||
m3u8_url, video_id, ext='mp4', m3u8_id='hls')
|
||||
|
||||
picture_paths = [
|
||||
episode_details.get('picture_path'),
|
||||
episode_details.get('large_picture_path'),
|
||||
]
|
||||
|
||||
thumbnails = [{
|
||||
'url': picture_path,
|
||||
'preference': idx,
|
||||
} for idx, picture_path in enumerate(picture_paths) if picture_path is not None]
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'title': episode_details['title'],
|
||||
'formats': formats,
|
||||
'thumbnails': thumbnails,
|
||||
'view_count': episode_details.get('view_count'),
|
||||
}
|
||||
@@ -14,13 +14,11 @@ from ..compat import (
|
||||
compat_urllib_parse_urlparse,
|
||||
)
|
||||
from ..utils import (
|
||||
determine_ext,
|
||||
ExtractorError,
|
||||
float_or_none,
|
||||
int_or_none,
|
||||
sanitized_Request,
|
||||
unsmuggle_url,
|
||||
update_url_query,
|
||||
xpath_with_ns,
|
||||
mimetype2ext,
|
||||
find_xpath_attr,
|
||||
@@ -50,12 +48,6 @@ class ThePlatformBaseIE(OnceIE):
|
||||
if OnceIE.suitable(_format['url']):
|
||||
formats.extend(self._extract_once_formats(_format['url']))
|
||||
else:
|
||||
media_url = _format['url']
|
||||
if determine_ext(media_url) == 'm3u8':
|
||||
hdnea2 = self._get_cookies(media_url).get('hdnea2')
|
||||
if hdnea2:
|
||||
_format['url'] = update_url_query(media_url, {'hdnea3': hdnea2.value})
|
||||
|
||||
formats.append(_format)
|
||||
|
||||
subtitles = self._parse_smil_subtitles(meta, default_ns)
|
||||
|
||||
@@ -12,7 +12,7 @@ class TheSixtyOneIE(InfoExtractor):
|
||||
s|
|
||||
song/comments/list|
|
||||
song
|
||||
)/(?:[^/]+/)?(?P<id>[A-Za-z0-9]+)/?$'''
|
||||
)/(?P<id>[A-Za-z0-9]+)/?$'''
|
||||
_SONG_URL_TEMPLATE = 'http://thesixtyone.com/s/{0:}'
|
||||
_SONG_FILE_URL_TEMPLATE = 'http://{audio_server:}/thesixtyone_production/audio/{0:}_stream'
|
||||
_THUMBNAIL_URL_TEMPLATE = '{photo_base_url:}_desktop'
|
||||
@@ -45,10 +45,6 @@ class TheSixtyOneIE(InfoExtractor):
|
||||
'url': 'http://www.thesixtyone.com/song/SrE3zD7s1jt/',
|
||||
'only_matching': True,
|
||||
},
|
||||
{
|
||||
'url': 'http://www.thesixtyone.com/maryatmidnight/song/StrawberriesandCream/yvWtLp0c4GQ/',
|
||||
'only_matching': True,
|
||||
},
|
||||
]
|
||||
|
||||
_DECODE_MAP = {
|
||||
|
||||
@@ -16,7 +16,6 @@ from ..compat import (
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
int_or_none,
|
||||
js_to_json,
|
||||
orderedSet,
|
||||
parse_duration,
|
||||
parse_iso8601,
|
||||
@@ -261,7 +260,7 @@ class TwitchVodIE(TwitchItemBaseIE):
|
||||
'nauth': access_token['token'],
|
||||
'nauthsig': access_token['sig'],
|
||||
})),
|
||||
item_id, 'mp4', entry_protocol='m3u8_native')
|
||||
item_id, 'mp4')
|
||||
|
||||
self._prefer_source(formats)
|
||||
info['formats'] = formats
|
||||
@@ -455,45 +454,3 @@ class TwitchStreamIE(TwitchBaseIE):
|
||||
'formats': formats,
|
||||
'is_live': True,
|
||||
}
|
||||
|
||||
|
||||
class TwitchClipsIE(InfoExtractor):
|
||||
IE_NAME = 'twitch:clips'
|
||||
_VALID_URL = r'https?://clips\.twitch\.tv/(?:[^/]+/)*(?P<id>[^/?#&]+)'
|
||||
|
||||
_TEST = {
|
||||
'url': 'https://clips.twitch.tv/ea/AggressiveCobraPoooound',
|
||||
'md5': '761769e1eafce0ffebfb4089cb3847cd',
|
||||
'info_dict': {
|
||||
'id': 'AggressiveCobraPoooound',
|
||||
'ext': 'mp4',
|
||||
'title': 'EA Play 2016 Live from the Novo Theatre',
|
||||
'thumbnail': 're:^https?://.*\.jpg',
|
||||
'creator': 'EA',
|
||||
'uploader': 'stereotype_',
|
||||
'uploader_id': 'stereotype_',
|
||||
},
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
clip = self._parse_json(
|
||||
self._search_regex(
|
||||
r'(?s)clipInfo\s*=\s*({.+?});', webpage, 'clip info'),
|
||||
video_id, transform_source=js_to_json)
|
||||
|
||||
video_url = clip['clip_video_url']
|
||||
title = clip['channel_title']
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'url': video_url,
|
||||
'title': title,
|
||||
'thumbnail': self._og_search_thumbnail(webpage),
|
||||
'creator': clip.get('broadcaster_display_name') or clip.get('broadcaster_login'),
|
||||
'uploader': clip.get('curator_login'),
|
||||
'uploader_id': clip.get('curator_display_name'),
|
||||
}
|
||||
|
||||
@@ -5,7 +5,6 @@ import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
determine_ext,
|
||||
float_or_none,
|
||||
xpath_text,
|
||||
remove_end,
|
||||
@@ -53,7 +52,7 @@ class TwitterCardIE(TwitterBaseIE):
|
||||
'id': 'dq4Oj5quskI',
|
||||
'ext': 'mp4',
|
||||
'title': 'Ubuntu 11.10 Overview',
|
||||
'description': 'Take a quick peek at what\'s new and improved in Ubuntu 11.10.\n\nOnce installed take a look at 10 Things to Do After Installing: http://www.omgubuntu.co.uk/2011/10/10...',
|
||||
'description': 'Take a quick peek at what\'s new and improved in Ubuntu 11.10.\n\nOnce installed take a look at 10 Things to Do After Installing: http://www.omgubuntu.co.uk/2011/10/10-things-to-do-after-installing-ubuntu-11-10/',
|
||||
'upload_date': '20111013',
|
||||
'uploader': 'OMG! Ubuntu!',
|
||||
'uploader_id': 'omgubuntu',
|
||||
@@ -117,16 +116,13 @@ class TwitterCardIE(TwitterBaseIE):
|
||||
video_url = config.get('video_url') or config.get('playlist', [{}])[0].get('source')
|
||||
|
||||
if video_url:
|
||||
if determine_ext(video_url) == 'm3u8':
|
||||
formats.extend(self._extract_m3u8_formats(video_url, video_id, ext='mp4', m3u8_id='hls'))
|
||||
else:
|
||||
f = {
|
||||
'url': video_url,
|
||||
}
|
||||
f = {
|
||||
'url': video_url,
|
||||
}
|
||||
|
||||
_search_dimensions_in_video_url(f, video_url)
|
||||
_search_dimensions_in_video_url(f, video_url)
|
||||
|
||||
formats.append(f)
|
||||
formats.append(f)
|
||||
|
||||
vmap_url = config.get('vmapUrl') or config.get('vmap_url')
|
||||
if vmap_url:
|
||||
@@ -211,7 +207,6 @@ class TwitterIE(InfoExtractor):
|
||||
'uploader_id': 'giphz',
|
||||
},
|
||||
'expected_warnings': ['height', 'width'],
|
||||
'skip': 'Account suspended',
|
||||
}, {
|
||||
'url': 'https://twitter.com/starwars/status/665052190608723968',
|
||||
'md5': '39b7199856dee6cd4432e72c74bc69d4',
|
||||
@@ -244,10 +239,10 @@ class TwitterIE(InfoExtractor):
|
||||
'info_dict': {
|
||||
'id': '700207533655363584',
|
||||
'ext': 'mp4',
|
||||
'title': 'Donte The Dumbass - BEAT PROD: @suhmeduh #Damndaniel',
|
||||
'description': 'Donte The Dumbass on Twitter: "BEAT PROD: @suhmeduh https://t.co/HBrQ4AfpvZ #Damndaniel https://t.co/byBooq2ejZ"',
|
||||
'title': 'jay - BEAT PROD: @suhmeduh #Damndaniel',
|
||||
'description': 'jay on Twitter: "BEAT PROD: @suhmeduh https://t.co/HBrQ4AfpvZ #Damndaniel https://t.co/byBooq2ejZ"',
|
||||
'thumbnail': 're:^https?://.*\.jpg',
|
||||
'uploader': 'Donte The Dumbass',
|
||||
'uploader': 'jay',
|
||||
'uploader_id': 'jaydingeer',
|
||||
},
|
||||
'params': {
|
||||
@@ -267,6 +262,7 @@ class TwitterIE(InfoExtractor):
|
||||
'add_ie': ['Vine'],
|
||||
}, {
|
||||
'url': 'https://twitter.com/captainamerica/status/719944021058060289',
|
||||
# md5 constantly changes
|
||||
'info_dict': {
|
||||
'id': '719944021058060289',
|
||||
'ext': 'mp4',
|
||||
@@ -275,9 +271,6 @@ class TwitterIE(InfoExtractor):
|
||||
'uploader_id': 'captainamerica',
|
||||
'uploader': 'Captain America',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True, # requires ffmpeg
|
||||
},
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
@@ -285,11 +278,7 @@ class TwitterIE(InfoExtractor):
|
||||
user_id = mobj.group('user_id')
|
||||
twid = mobj.group('id')
|
||||
|
||||
webpage, urlh = self._download_webpage_handle(
|
||||
self._TEMPLATE_URL % (user_id, twid), twid)
|
||||
|
||||
if 'twitter.com/account/suspended' in urlh.geturl():
|
||||
raise ExtractorError('Account suspended by Twitter.', expected=True)
|
||||
webpage = self._download_webpage(self._TEMPLATE_URL % (user_id, twid), twid)
|
||||
|
||||
username = remove_end(self._og_search_title(webpage), ' on Twitter')
|
||||
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
@@ -13,11 +12,11 @@ from ..utils import (
|
||||
|
||||
|
||||
class VesselIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?vessel\.com/(?:videos|embed)/(?P<id>[0-9a-zA-Z]+)'
|
||||
_VALID_URL = r'https?://(?:www\.)?vessel\.com/videos/(?P<id>[0-9a-zA-Z]+)'
|
||||
_API_URL_TEMPLATE = 'https://www.vessel.com/api/view/items/%s'
|
||||
_LOGIN_URL = 'https://www.vessel.com/api/account/login'
|
||||
_NETRC_MACHINE = 'vessel'
|
||||
_TESTS = [{
|
||||
_TEST = {
|
||||
'url': 'https://www.vessel.com/videos/HDN7G5UMs',
|
||||
'md5': '455cdf8beb71c6dd797fd2f3818d05c4',
|
||||
'info_dict': {
|
||||
@@ -29,16 +28,7 @@ class VesselIE(InfoExtractor):
|
||||
'description': 'Did Nvidia pull out all the stops on the Titan X, or does its performance leave something to be desired?',
|
||||
'timestamp': int,
|
||||
},
|
||||
}, {
|
||||
'url': 'https://www.vessel.com/embed/G4U7gUJ6a?w=615&h=346',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
@staticmethod
|
||||
def _extract_urls(webpage):
|
||||
return [url for _, url in re.findall(
|
||||
r'<iframe[^>]+src=(["\'])((?:https?:)?//(?:www\.)?vessel\.com/embed/[0-9a-zA-Z]+.*?)\1',
|
||||
webpage)]
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def make_json_request(url, data):
|
||||
@@ -108,24 +98,16 @@ class VesselIE(InfoExtractor):
|
||||
|
||||
formats = []
|
||||
for f in video_asset.get('sources', []):
|
||||
location = f.get('location')
|
||||
if not location:
|
||||
continue
|
||||
name = f.get('name')
|
||||
if name == 'hls-index':
|
||||
if f['name'] == 'hls-index':
|
||||
formats.extend(self._extract_m3u8_formats(
|
||||
location, video_id, ext='mp4',
|
||||
entry_protocol='m3u8_native', m3u8_id='m3u8', fatal=False))
|
||||
elif name == 'dash-index':
|
||||
formats.extend(self._extract_mpd_formats(
|
||||
location, video_id, mpd_id='dash', fatal=False))
|
||||
f['location'], video_id, ext='mp4', m3u8_id='m3u8'))
|
||||
else:
|
||||
formats.append({
|
||||
'format_id': name,
|
||||
'format_id': f['name'],
|
||||
'tbr': f.get('bitrate'),
|
||||
'height': f.get('height'),
|
||||
'width': f.get('width'),
|
||||
'url': location,
|
||||
'url': f['location'],
|
||||
})
|
||||
self._sort_formats(formats)
|
||||
|
||||
|
||||
@@ -1,73 +0,0 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import int_or_none
|
||||
|
||||
|
||||
class VidioIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?vidio\.com/watch/(?P<id>\d+)-(?P<display_id>[^/?#&]+)'
|
||||
_TESTS = [{
|
||||
'url': 'http://www.vidio.com/watch/165683-dj_ambred-booyah-live-2015',
|
||||
'md5': 'cd2801394afc164e9775db6a140b91fe',
|
||||
'info_dict': {
|
||||
'id': '165683',
|
||||
'display_id': 'dj_ambred-booyah-live-2015',
|
||||
'ext': 'mp4',
|
||||
'title': 'DJ_AMBRED - Booyah (Live 2015)',
|
||||
'description': 'md5:27dc15f819b6a78a626490881adbadf8',
|
||||
'thumbnail': 're:^https?://.*\.jpg$',
|
||||
'duration': 149,
|
||||
'like_count': int,
|
||||
},
|
||||
}, {
|
||||
'url': 'https://www.vidio.com/watch/77949-south-korea-test-fires-missile-that-can-strike-all-of-the-north',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
video_id, display_id = mobj.group('id', 'display_id')
|
||||
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
|
||||
title = self._og_search_title(webpage)
|
||||
|
||||
m3u8_url, duration, thumbnail = [None] * 3
|
||||
|
||||
clips = self._parse_json(
|
||||
self._html_search_regex(
|
||||
r'data-json-clips\s*=\s*(["\'])(?P<data>\[.+?\])\1',
|
||||
webpage, 'video data', default='[]', group='data'),
|
||||
display_id, fatal=False)
|
||||
if clips:
|
||||
clip = clips[0]
|
||||
m3u8_url = clip.get('sources', [{}])[0].get('file')
|
||||
duration = clip.get('clip_duration')
|
||||
thumbnail = clip.get('image')
|
||||
|
||||
m3u8_url = m3u8_url or self._search_regex(
|
||||
r'data(?:-vjs)?-clip-hls-url=(["\'])(?P<url>.+?)\1', webpage, 'hls url')
|
||||
formats = self._extract_m3u8_formats(m3u8_url, display_id, 'mp4', entry_protocol='m3u8_native')
|
||||
|
||||
duration = int_or_none(duration or self._search_regex(
|
||||
r'data-video-duration=(["\'])(?P<duartion>\d+)\1', webpage, 'duration'))
|
||||
thumbnail = thumbnail or self._og_search_thumbnail(webpage)
|
||||
|
||||
like_count = int_or_none(self._search_regex(
|
||||
(r'<span[^>]+data-comment-vote-count=["\'](\d+)',
|
||||
r'<span[^>]+class=["\'].*?\blike(?:__|-)count\b.*?["\'][^>]*>\s*(\d+)'),
|
||||
webpage, 'like count', fatal=False))
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'display_id': display_id,
|
||||
'title': title,
|
||||
'description': self._og_search_description(webpage),
|
||||
'thumbnail': thumbnail,
|
||||
'duration': duration,
|
||||
'like_count': like_count,
|
||||
'formats': formats,
|
||||
}
|
||||
@@ -101,13 +101,10 @@ class VikiBaseIE(InfoExtractor):
|
||||
self.report_warning('Unable to get session token, login has probably failed')
|
||||
|
||||
@staticmethod
|
||||
def dict_selection(dict_obj, preferred_key, allow_fallback=True):
|
||||
def dict_selection(dict_obj, preferred_key):
|
||||
if preferred_key in dict_obj:
|
||||
return dict_obj.get(preferred_key)
|
||||
|
||||
if not allow_fallback:
|
||||
return
|
||||
|
||||
filtered_dict = list(filter(None, [dict_obj.get(k) for k in dict_obj.keys()]))
|
||||
return filtered_dict[0] if filtered_dict else None
|
||||
|
||||
@@ -130,7 +127,7 @@ class VikiIE(VikiBaseIE):
|
||||
}, {
|
||||
# clip
|
||||
'url': 'http://www.viki.com/videos/1067139v-the-avengers-age-of-ultron-press-conference',
|
||||
'md5': 'feea2b1d7b3957f70886e6dfd8b8be84',
|
||||
'md5': '86c0b5dbd4d83a6611a79987cc7a1989',
|
||||
'info_dict': {
|
||||
'id': '1067139v',
|
||||
'ext': 'mp4',
|
||||
@@ -159,18 +156,17 @@ class VikiIE(VikiBaseIE):
|
||||
'params': {
|
||||
# m3u8 download
|
||||
'skip_download': True,
|
||||
},
|
||||
'skip': 'Blocked in the US',
|
||||
}
|
||||
}, {
|
||||
# episode
|
||||
'url': 'http://www.viki.com/videos/44699v-boys-over-flowers-episode-1',
|
||||
'md5': '1f54697dabc8f13f31bf06bb2e4de6db',
|
||||
'md5': '190f3ef426005ba3a080a63325955bc3',
|
||||
'info_dict': {
|
||||
'id': '44699v',
|
||||
'ext': 'mp4',
|
||||
'title': 'Boys Over Flowers - Episode 1',
|
||||
'description': 'md5:b89cf50038b480b88b5b3c93589a9076',
|
||||
'duration': 4204,
|
||||
'description': 'md5:52617e4f729c7d03bfd4bcbbb6e946f2',
|
||||
'duration': 4155,
|
||||
'timestamp': 1270496524,
|
||||
'upload_date': '20100405',
|
||||
'uploader': 'group8',
|
||||
@@ -200,7 +196,7 @@ class VikiIE(VikiBaseIE):
|
||||
}, {
|
||||
# non-English description
|
||||
'url': 'http://www.viki.com/videos/158036v-love-in-magic',
|
||||
'md5': '013dc282714e22acf9447cad14ff1208',
|
||||
'md5': '1713ae35df5a521b31f6dc40730e7c9c',
|
||||
'info_dict': {
|
||||
'id': '158036v',
|
||||
'ext': 'mp4',
|
||||
@@ -221,7 +217,7 @@ class VikiIE(VikiBaseIE):
|
||||
|
||||
self._check_errors(video)
|
||||
|
||||
title = self.dict_selection(video.get('titles', {}), 'en', allow_fallback=False)
|
||||
title = self.dict_selection(video.get('titles', {}), 'en')
|
||||
if not title:
|
||||
title = 'Episode %d' % video.get('number') if video.get('type') == 'episode' else video.get('id') or video_id
|
||||
container_titles = video.get('container', {}).get('titles', {})
|
||||
@@ -306,7 +302,7 @@ class VikiChannelIE(VikiBaseIE):
|
||||
'title': 'Boys Over Flowers',
|
||||
'description': 'md5:ecd3cff47967fe193cff37c0bec52790',
|
||||
},
|
||||
'playlist_mincount': 71,
|
||||
'playlist_count': 70,
|
||||
}, {
|
||||
'url': 'http://www.viki.com/tv/1354c-poor-nastya-complete',
|
||||
'info_dict': {
|
||||
|
||||
@@ -8,7 +8,6 @@ import itertools
|
||||
from .common import InfoExtractor
|
||||
from ..compat import (
|
||||
compat_HTTPError,
|
||||
compat_str,
|
||||
compat_urlparse,
|
||||
)
|
||||
from ..utils import (
|
||||
@@ -25,7 +24,6 @@ from ..utils import (
|
||||
urlencode_postdata,
|
||||
unescapeHTML,
|
||||
parse_filesize,
|
||||
try_get,
|
||||
)
|
||||
|
||||
|
||||
@@ -68,69 +66,6 @@ class VimeoBaseInfoExtractor(InfoExtractor):
|
||||
def _set_vimeo_cookie(self, name, value):
|
||||
self._set_cookie('vimeo.com', name, value)
|
||||
|
||||
def _vimeo_sort_formats(self, formats):
|
||||
# Bitrates are completely broken. Single m3u8 may contain entries in kbps and bps
|
||||
# at the same time without actual units specified. This lead to wrong sorting.
|
||||
self._sort_formats(formats, field_preference=('preference', 'height', 'width', 'fps', 'format_id'))
|
||||
|
||||
def _parse_config(self, config, video_id):
|
||||
# Extract title
|
||||
video_title = config['video']['title']
|
||||
|
||||
# Extract uploader, uploader_url and uploader_id
|
||||
video_uploader = config['video'].get('owner', {}).get('name')
|
||||
video_uploader_url = config['video'].get('owner', {}).get('url')
|
||||
video_uploader_id = video_uploader_url.split('/')[-1] if video_uploader_url else None
|
||||
|
||||
# Extract video thumbnail
|
||||
video_thumbnail = config['video'].get('thumbnail')
|
||||
if video_thumbnail is None:
|
||||
video_thumbs = config['video'].get('thumbs')
|
||||
if video_thumbs and isinstance(video_thumbs, dict):
|
||||
_, video_thumbnail = sorted((int(width if width.isdigit() else 0), t_url) for (width, t_url) in video_thumbs.items())[-1]
|
||||
|
||||
# Extract video duration
|
||||
video_duration = int_or_none(config['video'].get('duration'))
|
||||
|
||||
formats = []
|
||||
config_files = config['video'].get('files') or config['request'].get('files', {})
|
||||
for f in config_files.get('progressive', []):
|
||||
video_url = f.get('url')
|
||||
if not video_url:
|
||||
continue
|
||||
formats.append({
|
||||
'url': video_url,
|
||||
'format_id': 'http-%s' % f.get('quality'),
|
||||
'width': int_or_none(f.get('width')),
|
||||
'height': int_or_none(f.get('height')),
|
||||
'fps': int_or_none(f.get('fps')),
|
||||
'tbr': int_or_none(f.get('bitrate')),
|
||||
})
|
||||
m3u8_url = config_files.get('hls', {}).get('url')
|
||||
if m3u8_url:
|
||||
formats.extend(self._extract_m3u8_formats(
|
||||
m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))
|
||||
|
||||
subtitles = {}
|
||||
text_tracks = config['request'].get('text_tracks')
|
||||
if text_tracks:
|
||||
for tt in text_tracks:
|
||||
subtitles[tt['lang']] = [{
|
||||
'ext': 'vtt',
|
||||
'url': 'https://vimeo.com' + tt['url'],
|
||||
}]
|
||||
|
||||
return {
|
||||
'title': video_title,
|
||||
'uploader': video_uploader,
|
||||
'uploader_id': video_uploader_id,
|
||||
'uploader_url': video_uploader_url,
|
||||
'thumbnail': video_thumbnail,
|
||||
'duration': video_duration,
|
||||
'formats': formats,
|
||||
'subtitles': subtitles,
|
||||
}
|
||||
|
||||
|
||||
class VimeoIE(VimeoBaseInfoExtractor):
|
||||
"""Information extractor for vimeo.com."""
|
||||
@@ -218,7 +153,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
|
||||
'uploader_id': 'user18948128',
|
||||
'uploader': 'Jaime Marquínez Ferrándiz',
|
||||
'duration': 10,
|
||||
'description': 'This is "youtube-dl password protected test video" by on Vimeo, the home for high quality videos and the people who love them.',
|
||||
'description': 'This is "youtube-dl password protected test video" by Jaime Marquínez Ferrándiz on Vimeo, the home for high quality videos and the people\u2026',
|
||||
},
|
||||
'params': {
|
||||
'videopassword': 'youtube-dl',
|
||||
@@ -447,24 +382,28 @@ class VimeoIE(VimeoBaseInfoExtractor):
|
||||
if config.get('view') == 4:
|
||||
config = self._verify_player_video_password(url, video_id)
|
||||
|
||||
def is_rented():
|
||||
if '>You rented this title.<' in webpage:
|
||||
return True
|
||||
if config.get('user', {}).get('purchased'):
|
||||
return True
|
||||
label = try_get(
|
||||
config, lambda x: x['video']['vod']['purchase_options'][0]['label_string'], compat_str)
|
||||
if label and label.startswith('You rented this'):
|
||||
return True
|
||||
return False
|
||||
|
||||
if is_rented():
|
||||
if '>You rented this title.<' in webpage:
|
||||
feature_id = config.get('video', {}).get('vod', {}).get('feature_id')
|
||||
if feature_id and not data.get('force_feature_id', False):
|
||||
return self.url_result(smuggle_url(
|
||||
'https://player.vimeo.com/player/%s' % feature_id,
|
||||
{'force_feature_id': True}), 'Vimeo')
|
||||
|
||||
# Extract title
|
||||
video_title = config['video']['title']
|
||||
|
||||
# Extract uploader, uploader_url and uploader_id
|
||||
video_uploader = config['video'].get('owner', {}).get('name')
|
||||
video_uploader_url = config['video'].get('owner', {}).get('url')
|
||||
video_uploader_id = video_uploader_url.split('/')[-1] if video_uploader_url else None
|
||||
|
||||
# Extract video thumbnail
|
||||
video_thumbnail = config['video'].get('thumbnail')
|
||||
if video_thumbnail is None:
|
||||
video_thumbs = config['video'].get('thumbs')
|
||||
if video_thumbs and isinstance(video_thumbs, dict):
|
||||
_, video_thumbnail = sorted((int(width if width.isdigit() else 0), t_url) for (width, t_url) in video_thumbs.items())[-1]
|
||||
|
||||
# Extract video description
|
||||
|
||||
video_description = self._html_search_regex(
|
||||
@@ -484,6 +423,9 @@ class VimeoIE(VimeoBaseInfoExtractor):
|
||||
if not video_description and not mobj.group('player'):
|
||||
self._downloader.report_warning('Cannot find video description')
|
||||
|
||||
# Extract video duration
|
||||
video_duration = int_or_none(config['video'].get('duration'))
|
||||
|
||||
# Extract upload date
|
||||
video_upload_date = None
|
||||
mobj = re.search(r'<time[^>]+datetime="([^"]+)"', webpage)
|
||||
@@ -521,22 +463,53 @@ class VimeoIE(VimeoBaseInfoExtractor):
|
||||
'format_id': source_name,
|
||||
'preference': 1,
|
||||
})
|
||||
config_files = config['video'].get('files') or config['request'].get('files', {})
|
||||
for f in config_files.get('progressive', []):
|
||||
video_url = f.get('url')
|
||||
if not video_url:
|
||||
continue
|
||||
formats.append({
|
||||
'url': video_url,
|
||||
'format_id': 'http-%s' % f.get('quality'),
|
||||
'width': int_or_none(f.get('width')),
|
||||
'height': int_or_none(f.get('height')),
|
||||
'fps': int_or_none(f.get('fps')),
|
||||
'tbr': int_or_none(f.get('bitrate')),
|
||||
})
|
||||
m3u8_url = config_files.get('hls', {}).get('url')
|
||||
if m3u8_url:
|
||||
formats.extend(self._extract_m3u8_formats(
|
||||
m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))
|
||||
# Bitrates are completely broken. Single m3u8 may contain entries in kbps and bps
|
||||
# at the same time without actual units specified. This lead to wrong sorting.
|
||||
self._sort_formats(formats, field_preference=('preference', 'height', 'width', 'fps', 'format_id'))
|
||||
|
||||
info_dict = self._parse_config(config, video_id)
|
||||
formats.extend(info_dict['formats'])
|
||||
self._vimeo_sort_formats(formats)
|
||||
info_dict.update({
|
||||
subtitles = {}
|
||||
text_tracks = config['request'].get('text_tracks')
|
||||
if text_tracks:
|
||||
for tt in text_tracks:
|
||||
subtitles[tt['lang']] = [{
|
||||
'ext': 'vtt',
|
||||
'url': 'https://vimeo.com' + tt['url'],
|
||||
}]
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'formats': formats,
|
||||
'uploader': video_uploader,
|
||||
'uploader_url': video_uploader_url,
|
||||
'uploader_id': video_uploader_id,
|
||||
'upload_date': video_upload_date,
|
||||
'title': video_title,
|
||||
'thumbnail': video_thumbnail,
|
||||
'description': video_description,
|
||||
'duration': video_duration,
|
||||
'formats': formats,
|
||||
'webpage_url': url,
|
||||
'view_count': view_count,
|
||||
'like_count': like_count,
|
||||
'comment_count': comment_count,
|
||||
})
|
||||
|
||||
return info_dict
|
||||
'subtitles': subtitles,
|
||||
}
|
||||
|
||||
|
||||
class VimeoOndemandIE(VimeoBaseInfoExtractor):
|
||||
@@ -719,7 +692,7 @@ class VimeoGroupsIE(VimeoAlbumIE):
|
||||
return self._extract_videos(name, 'https://vimeo.com/groups/%s' % name)
|
||||
|
||||
|
||||
class VimeoReviewIE(VimeoBaseInfoExtractor):
|
||||
class VimeoReviewIE(InfoExtractor):
|
||||
IE_NAME = 'vimeo:review'
|
||||
IE_DESC = 'Review pages on vimeo'
|
||||
_VALID_URL = r'https://vimeo\.com/[^/]+/review/(?P<id>[^/]+)'
|
||||
@@ -731,7 +704,6 @@ class VimeoReviewIE(VimeoBaseInfoExtractor):
|
||||
'ext': 'mp4',
|
||||
'title': "DICK HARDWICK 'Comedian'",
|
||||
'uploader': 'Richard Hardwick',
|
||||
'uploader_id': 'user21297594',
|
||||
}
|
||||
}, {
|
||||
'note': 'video player needs Referer',
|
||||
@@ -744,18 +716,14 @@ class VimeoReviewIE(VimeoBaseInfoExtractor):
|
||||
'uploader': 'DevWeek Events',
|
||||
'duration': 2773,
|
||||
'thumbnail': 're:^https?://.*\.jpg$',
|
||||
'uploader_id': 'user22258446',
|
||||
}
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
config = self._download_json(
|
||||
'https://player.vimeo.com/video/%s/config' % video_id, video_id)
|
||||
info_dict = self._parse_config(config, video_id)
|
||||
self._vimeo_sort_formats(info_dict['formats'])
|
||||
info_dict['id'] = video_id
|
||||
return info_dict
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
video_id = mobj.group('id')
|
||||
player_url = 'https://player.vimeo.com/player/' + video_id
|
||||
return self.url_result(player_url, 'Vimeo', video_id)
|
||||
|
||||
|
||||
class VimeoWatchLaterIE(VimeoChannelIE):
|
||||
|
||||
@@ -9,7 +9,6 @@ from ..utils import (
|
||||
ExtractorError,
|
||||
float_or_none,
|
||||
int_or_none,
|
||||
remove_start,
|
||||
)
|
||||
from ..compat import compat_urllib_parse_urlencode
|
||||
|
||||
@@ -40,7 +39,6 @@ class VLiveIE(InfoExtractor):
|
||||
webpage, 'video params')
|
||||
status, _, _, live_params, long_video_id, key = re.split(
|
||||
r'"\s*,\s*"', video_params)[2:8]
|
||||
status = remove_start(status, 'PRODUCT_')
|
||||
|
||||
if status == 'LIVE_ON_AIR' or status == 'BIG_EVENT_ON_AIR':
|
||||
live_params = self._parse_json('"%s"' % live_params, video_id)
|
||||
|
||||
@@ -3,10 +3,7 @@ from __future__ import unicode_literals
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..compat import (
|
||||
compat_str,
|
||||
compat_urlparse,
|
||||
)
|
||||
from ..compat import compat_urlparse
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
determine_ext,
|
||||
@@ -19,13 +16,13 @@ class VoiceRepublicIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://voicerepublic\.com/(?:talks|embed)/(?P<id>[0-9a-z-]+)'
|
||||
_TESTS = [{
|
||||
'url': 'http://voicerepublic.com/talks/watching-the-watchers-building-a-sousveillance-state',
|
||||
'md5': 'b9174d651323f17783000876347116e3',
|
||||
'md5': '0554a24d1657915aa8e8f84e15dc9353',
|
||||
'info_dict': {
|
||||
'id': '2296',
|
||||
'display_id': 'watching-the-watchers-building-a-sousveillance-state',
|
||||
'ext': 'm4a',
|
||||
'title': 'Watching the Watchers: Building a Sousveillance State',
|
||||
'description': 'Secret surveillance programs have metadata too. The people and companies that operate secret surveillance programs can be surveilled.',
|
||||
'description': 'md5:715ba964958afa2398df615809cfecb1',
|
||||
'thumbnail': 're:^https?://.*\.(?:png|jpg)$',
|
||||
'duration': 1800,
|
||||
'view_count': int,
|
||||
@@ -55,7 +52,7 @@ class VoiceRepublicIE(InfoExtractor):
|
||||
if data:
|
||||
title = data['title']
|
||||
description = data.get('teaser')
|
||||
talk_id = compat_str(data.get('talk_id') or display_id)
|
||||
talk_id = data.get('talk_id') or display_id
|
||||
talk = data['talk']
|
||||
duration = int_or_none(talk.get('duration'))
|
||||
formats = [{
|
||||
|
||||
@@ -4,7 +4,6 @@ import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
parse_duration,
|
||||
str_to_int,
|
||||
)
|
||||
@@ -28,8 +27,7 @@ class VpornIE(InfoExtractor):
|
||||
'duration': 393,
|
||||
'age_limit': 18,
|
||||
'view_count': int,
|
||||
},
|
||||
'skip': 'video removed',
|
||||
}
|
||||
},
|
||||
{
|
||||
'url': 'http://www.vporn.com/female/hana-shower/523564/',
|
||||
@@ -42,7 +40,7 @@ class VpornIE(InfoExtractor):
|
||||
'description': 'Hana showers at the bathroom.',
|
||||
'thumbnail': 're:^https?://.*\.jpg$',
|
||||
'uploader': 'Hmmmmm',
|
||||
'categories': ['Big Boobs', 'Erotic', 'Teen', 'Female', '720p'],
|
||||
'categories': ['Big Boobs', 'Erotic', 'Teen', 'Female'],
|
||||
'duration': 588,
|
||||
'age_limit': 18,
|
||||
'view_count': int,
|
||||
@@ -57,10 +55,6 @@ class VpornIE(InfoExtractor):
|
||||
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
|
||||
errmsg = 'This video has been deleted due to Copyright Infringement or by the account owner!'
|
||||
if errmsg in webpage:
|
||||
raise ExtractorError('%s said: %s' % (self.IE_NAME, errmsg), expected=True)
|
||||
|
||||
title = self._html_search_regex(
|
||||
r'videoname\s*=\s*\'([^\']+)\'', webpage, 'title').strip()
|
||||
description = self._html_search_regex(
|
||||
|
||||
69
youtube_dl/extractor/vulture.py
Normal file
69
youtube_dl/extractor/vulture.py
Normal file
@@ -0,0 +1,69 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import json
|
||||
import os.path
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
int_or_none,
|
||||
parse_iso8601,
|
||||
)
|
||||
|
||||
|
||||
class VultureIE(InfoExtractor):
|
||||
IE_NAME = 'vulture.com'
|
||||
_VALID_URL = r'https?://video\.vulture\.com/video/(?P<display_id>[^/]+)/'
|
||||
_TEST = {
|
||||
'url': 'http://video.vulture.com/video/Mindy-Kaling-s-Harvard-Speech/player?layout=compact&read_more=1',
|
||||
'md5': '8d997845642a2b5152820f7257871bc8',
|
||||
'info_dict': {
|
||||
'id': '6GHRQL3RV7MSD1H4',
|
||||
'ext': 'mp4',
|
||||
'title': 'kaling-speech-2-MAGNIFY STANDARD CONTAINER REVISED',
|
||||
'uploader_id': 'Sarah',
|
||||
'thumbnail': 're:^http://.*\.jpg$',
|
||||
'timestamp': 1401288564,
|
||||
'upload_date': '20140528',
|
||||
'description': 'Uplifting and witty, as predicted.',
|
||||
'duration': 1015,
|
||||
}
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
display_id = mobj.group('display_id')
|
||||
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
query_string = self._search_regex(
|
||||
r"queryString\s*=\s*'([^']+)'", webpage, 'query string')
|
||||
video_id = self._search_regex(
|
||||
r'content=([^&]+)', query_string, 'video ID')
|
||||
query_url = 'http://video.vulture.com/embed/player/container/1000/1000/?%s' % query_string
|
||||
|
||||
query_webpage = self._download_webpage(
|
||||
query_url, display_id, note='Downloading query page')
|
||||
params_json = self._search_regex(
|
||||
r'(?sm)new MagnifyEmbeddablePlayer\({.*?contentItem:\s*(\{.*?\})\n?,\n',
|
||||
query_webpage,
|
||||
'player params')
|
||||
params = json.loads(params_json)
|
||||
|
||||
upload_timestamp = parse_iso8601(params['posted'].replace(' ', 'T'))
|
||||
uploader_id = params.get('user', {}).get('handle')
|
||||
|
||||
media_item = params['media_item']
|
||||
title = os.path.splitext(media_item['title'])[0]
|
||||
duration = int_or_none(media_item.get('duration_seconds'))
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'display_id': display_id,
|
||||
'url': media_item['pipeline_xid'],
|
||||
'title': title,
|
||||
'timestamp': upload_timestamp,
|
||||
'thumbnail': params.get('thumbnail_url'),
|
||||
'uploader_id': uploader_id,
|
||||
'description': params.get('description'),
|
||||
'duration': duration,
|
||||
}
|
||||
@@ -1,236 +1,215 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import itertools
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..compat import (
|
||||
compat_parse_qs,
|
||||
compat_urlparse,
|
||||
)
|
||||
from ..utils import (
|
||||
determine_ext,
|
||||
ExtractorError,
|
||||
js_to_json,
|
||||
strip_jsonp,
|
||||
unified_strdate,
|
||||
update_url_query,
|
||||
urlhandle_detect_ext,
|
||||
qualities,
|
||||
)
|
||||
|
||||
|
||||
class WDRBaseIE(InfoExtractor):
|
||||
def _extract_wdr_video(self, webpage, display_id):
|
||||
# for wdr.de the data-extension is in a tag with the class "mediaLink"
|
||||
# for wdr.de radio players, in a tag with the class "wdrrPlayerPlayBtn"
|
||||
# for wdrmaus its in a link to the page in a multiline "videoLink"-tag
|
||||
json_metadata = self._html_search_regex(
|
||||
r'class=(?:"(?:mediaLink|wdrrPlayerPlayBtn)\b[^"]*"[^>]+|"videoLink\b[^"]*"[\s]*>\n[^\n]*)data-extension="([^"]+)"',
|
||||
webpage, 'media link', default=None, flags=re.MULTILINE)
|
||||
|
||||
if not json_metadata:
|
||||
return
|
||||
|
||||
media_link_obj = self._parse_json(json_metadata, display_id,
|
||||
transform_source=js_to_json)
|
||||
jsonp_url = media_link_obj['mediaObj']['url']
|
||||
|
||||
metadata = self._download_json(
|
||||
jsonp_url, 'metadata', transform_source=strip_jsonp)
|
||||
|
||||
metadata_tracker_data = metadata['trackerData']
|
||||
metadata_media_resource = metadata['mediaResource']
|
||||
|
||||
formats = []
|
||||
|
||||
# check if the metadata contains a direct URL to a file
|
||||
for kind, media_resource in metadata_media_resource.items():
|
||||
if kind not in ('dflt', 'alt'):
|
||||
continue
|
||||
|
||||
for tag_name, medium_url in media_resource.items():
|
||||
if tag_name not in ('videoURL', 'audioURL'):
|
||||
continue
|
||||
|
||||
ext = determine_ext(medium_url)
|
||||
if ext == 'm3u8':
|
||||
formats.extend(self._extract_m3u8_formats(
|
||||
medium_url, display_id, 'mp4', 'm3u8_native',
|
||||
m3u8_id='hls'))
|
||||
elif ext == 'f4m':
|
||||
manifest_url = update_url_query(
|
||||
medium_url, {'hdcore': '3.2.0', 'plugin': 'aasp-3.2.0.77.18'})
|
||||
formats.extend(self._extract_f4m_formats(
|
||||
manifest_url, display_id, f4m_id='hds', fatal=False))
|
||||
elif ext == 'smil':
|
||||
formats.extend(self._extract_smil_formats(
|
||||
medium_url, 'stream', fatal=False))
|
||||
else:
|
||||
a_format = {
|
||||
'url': medium_url
|
||||
}
|
||||
if ext == 'unknown_video':
|
||||
urlh = self._request_webpage(
|
||||
medium_url, display_id, note='Determining extension')
|
||||
ext = urlhandle_detect_ext(urlh)
|
||||
a_format['ext'] = ext
|
||||
formats.append(a_format)
|
||||
|
||||
self._sort_formats(formats)
|
||||
|
||||
subtitles = {}
|
||||
caption_url = metadata_media_resource.get('captionURL')
|
||||
if caption_url:
|
||||
subtitles['de'] = [{
|
||||
'url': caption_url,
|
||||
'ext': 'ttml',
|
||||
}]
|
||||
|
||||
title = metadata_tracker_data['trackerClipTitle']
|
||||
|
||||
return {
|
||||
'id': metadata_tracker_data.get('trackerClipId', display_id),
|
||||
'display_id': display_id,
|
||||
'title': title,
|
||||
'alt_title': metadata_tracker_data.get('trackerClipSubcategory'),
|
||||
'formats': formats,
|
||||
'subtitles': subtitles,
|
||||
'upload_date': unified_strdate(metadata_tracker_data.get('trackerClipAirTime')),
|
||||
}
|
||||
|
||||
|
||||
class WDRIE(WDRBaseIE):
|
||||
_CURRENT_MAUS_URL = r'https?://(?:www\.)wdrmaus.de/(?:[^/]+/){1,2}[^/?#]+\.php5'
|
||||
_PAGE_REGEX = r'/(?:mediathek/)?[^/]+/(?P<type>[^/]+)/(?P<display_id>.+)\.html'
|
||||
_VALID_URL = r'(?P<page_url>https?://(?:www\d\.)?wdr\d?\.de)' + _PAGE_REGEX + '|' + _CURRENT_MAUS_URL
|
||||
class WDRIE(InfoExtractor):
|
||||
_PLAYER_REGEX = '-(?:video|audio)player(?:_size-[LMS])?'
|
||||
_VALID_URL = r'(?P<url>https?://www\d?\.(?:wdr\d?|funkhauseuropa)\.de/)(?P<id>.+?)(?P<player>%s)?\.html' % _PLAYER_REGEX
|
||||
|
||||
_TESTS = [
|
||||
{
|
||||
'url': 'http://www1.wdr.de/mediathek/video/sendungen/doku-am-freitag/video-geheimnis-aachener-dom-100.html',
|
||||
# HDS download, MD5 is unstable
|
||||
'url': 'http://www1.wdr.de/mediathek/video/sendungen/servicezeit/videoservicezeit560-videoplayer_size-L.html',
|
||||
'info_dict': {
|
||||
'id': 'mdb-1058683',
|
||||
'id': 'mdb-362427',
|
||||
'ext': 'flv',
|
||||
'display_id': 'doku-am-freitag/video-geheimnis-aachener-dom-100',
|
||||
'title': 'Geheimnis Aachener Dom',
|
||||
'alt_title': 'Doku am Freitag',
|
||||
'upload_date': '20160304',
|
||||
'description': 'md5:87be8ff14d8dfd7a7ee46f0299b52318',
|
||||
'is_live': False,
|
||||
'subtitles': {'de': [{
|
||||
'url': 'http://ondemand-ww.wdr.de/medp/fsk0/105/1058683/1058683_12220974.xml',
|
||||
'ext': 'ttml',
|
||||
}]},
|
||||
},
|
||||
},
|
||||
{
|
||||
'url': 'http://www1.wdr.de/mediathek/audio/wdr3/wdr3-gespraech-am-samstag/audio-schriftstellerin-juli-zeh-100.html',
|
||||
'md5': 'f4c1f96d01cf285240f53ea4309663d8',
|
||||
'info_dict': {
|
||||
'id': 'mdb-1072000',
|
||||
'ext': 'mp3',
|
||||
'display_id': 'wdr3-gespraech-am-samstag/audio-schriftstellerin-juli-zeh-100',
|
||||
'title': 'Schriftstellerin Juli Zeh',
|
||||
'alt_title': 'WDR 3 Gespräch am Samstag',
|
||||
'upload_date': '20160312',
|
||||
'description': 'md5:e127d320bc2b1f149be697ce044a3dd7',
|
||||
'is_live': False,
|
||||
'subtitles': {}
|
||||
},
|
||||
},
|
||||
{
|
||||
'url': 'http://www1.wdr.de/mediathek/video/live/index.html',
|
||||
'info_dict': {
|
||||
'id': 'mdb-103364',
|
||||
'ext': 'mp4',
|
||||
'display_id': 'index',
|
||||
'title': r're:^WDR Fernsehen im Livestream [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
|
||||
'alt_title': 'WDR Fernsehen Live',
|
||||
'upload_date': None,
|
||||
'description': 'md5:ae2ff888510623bf8d4b115f95a9b7c9',
|
||||
'is_live': True,
|
||||
'subtitles': {}
|
||||
'title': 'Servicezeit',
|
||||
'description': 'md5:c8f43e5e815eeb54d0b96df2fba906cb',
|
||||
'upload_date': '20140310',
|
||||
'is_live': False
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True, # m3u8 download
|
||||
'skip_download': True,
|
||||
},
|
||||
'skip': 'Page Not Found',
|
||||
},
|
||||
{
|
||||
'url': 'http://www1.wdr.de/mediathek/video/sendungen/aktuelle-stunde/aktuelle-stunde-120.html',
|
||||
'playlist_mincount': 8,
|
||||
'url': 'http://www1.wdr.de/themen/av/videomargaspiegelisttot101-videoplayer.html',
|
||||
'info_dict': {
|
||||
'id': 'aktuelle-stunde/aktuelle-stunde-120',
|
||||
},
|
||||
},
|
||||
{
|
||||
'url': 'http://www.wdrmaus.de/aktuelle-sendung/index.php5',
|
||||
'info_dict': {
|
||||
'id': 'mdb-1096487',
|
||||
'id': 'mdb-363194',
|
||||
'ext': 'flv',
|
||||
'upload_date': 're:^[0-9]{8}$',
|
||||
'title': 're:^Die Sendung mit der Maus vom [0-9.]{10}$',
|
||||
'description': '- Die Sendung mit der Maus -',
|
||||
},
|
||||
'skip': 'The id changes from week to week because of the new episode'
|
||||
},
|
||||
{
|
||||
'url': 'http://www.wdrmaus.de/sachgeschichten/sachgeschichten/achterbahn.php5',
|
||||
'md5': '803138901f6368ee497b4d195bb164f2',
|
||||
'info_dict': {
|
||||
'id': 'mdb-186083',
|
||||
'ext': 'mp4',
|
||||
'upload_date': '20130919',
|
||||
'title': 'Sachgeschichte - Achterbahn ',
|
||||
'description': '- Die Sendung mit der Maus -',
|
||||
},
|
||||
},
|
||||
{
|
||||
'url': 'http://www1.wdr.de/radio/player/radioplayer116~_layout-popupVersion.html',
|
||||
# Live stream, MD5 unstable
|
||||
'info_dict': {
|
||||
'id': 'mdb-869971',
|
||||
'ext': 'flv',
|
||||
'title': 'Funkhaus Europa Livestream',
|
||||
'title': 'Marga Spiegel ist tot',
|
||||
'description': 'md5:2309992a6716c347891c045be50992e4',
|
||||
'upload_date': '20160101',
|
||||
'upload_date': '20140311',
|
||||
'is_live': False
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
'skip': 'Page Not Found',
|
||||
},
|
||||
{
|
||||
'url': 'http://www1.wdr.de/themen/kultur/audioerlebtegeschichtenmargaspiegel100-audioplayer.html',
|
||||
'md5': '83e9e8fefad36f357278759870805898',
|
||||
'info_dict': {
|
||||
'id': 'mdb-194332',
|
||||
'ext': 'mp3',
|
||||
'title': 'Erlebte Geschichten: Marga Spiegel (29.11.2009)',
|
||||
'description': 'md5:2309992a6716c347891c045be50992e4',
|
||||
'upload_date': '20091129',
|
||||
'is_live': False
|
||||
},
|
||||
},
|
||||
{
|
||||
'url': 'http://www.funkhauseuropa.de/av/audioflaviacoelhoamaramar100-audioplayer.html',
|
||||
'md5': '99a1443ff29af19f6c52cf6f4dc1f4aa',
|
||||
'info_dict': {
|
||||
'id': 'mdb-478135',
|
||||
'ext': 'mp3',
|
||||
'title': 'Flavia Coelho: Amar é Amar',
|
||||
'description': 'md5:7b29e97e10dfb6e265238b32fa35b23a',
|
||||
'upload_date': '20140717',
|
||||
'is_live': False
|
||||
},
|
||||
'skip': 'Page Not Found',
|
||||
},
|
||||
{
|
||||
'url': 'http://www1.wdr.de/mediathek/video/sendungen/quarks_und_co/filterseite-quarks-und-co100.html',
|
||||
'playlist_mincount': 146,
|
||||
'info_dict': {
|
||||
'id': 'mediathek/video/sendungen/quarks_und_co/filterseite-quarks-und-co100',
|
||||
}
|
||||
},
|
||||
{
|
||||
'url': 'http://www1.wdr.de/mediathek/video/livestream/index.html',
|
||||
'info_dict': {
|
||||
'id': 'mdb-103364',
|
||||
'title': 're:^WDR Fernsehen Live [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
|
||||
'description': 'md5:ae2ff888510623bf8d4b115f95a9b7c9',
|
||||
'ext': 'flv',
|
||||
'upload_date': '20150101',
|
||||
'is_live': True
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
url_type = mobj.group('type')
|
||||
page_url = mobj.group('page_url')
|
||||
display_id = mobj.group('display_id')
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
page_url = mobj.group('url')
|
||||
page_id = mobj.group('id')
|
||||
|
||||
info_dict = self._extract_wdr_video(webpage, display_id)
|
||||
webpage = self._download_webpage(url, page_id)
|
||||
|
||||
if not info_dict:
|
||||
if mobj.group('player') is None:
|
||||
entries = [
|
||||
self.url_result(page_url + href[0], 'WDR')
|
||||
self.url_result(page_url + href, 'WDR')
|
||||
for href in re.findall(
|
||||
r'<a href="(%s)"[^>]+data-extension=' % self._PAGE_REGEX,
|
||||
r'<a href="/?(.+?%s\.html)" rel="nofollow"' % self._PLAYER_REGEX,
|
||||
webpage)
|
||||
]
|
||||
|
||||
if entries: # Playlist page
|
||||
return self.playlist_result(entries, playlist_id=display_id)
|
||||
return self.playlist_result(entries, page_id)
|
||||
|
||||
raise ExtractorError('No downloadable streams found', expected=True)
|
||||
# Overview page
|
||||
entries = []
|
||||
for page_num in itertools.count(2):
|
||||
hrefs = re.findall(
|
||||
r'<li class="mediathekvideo"\s*>\s*<img[^>]*>\s*<a href="(/mediathek/video/[^"]+)"',
|
||||
webpage)
|
||||
entries.extend(
|
||||
self.url_result(page_url + href, 'WDR')
|
||||
for href in hrefs)
|
||||
next_url_m = re.search(
|
||||
r'<li class="nextToLast">\s*<a href="([^"]+)"', webpage)
|
||||
if not next_url_m:
|
||||
break
|
||||
next_url = page_url + next_url_m.group(1)
|
||||
webpage = self._download_webpage(
|
||||
next_url, page_id,
|
||||
note='Downloading playlist page %d' % page_num)
|
||||
return self.playlist_result(entries, page_id)
|
||||
|
||||
is_live = url_type == 'live'
|
||||
flashvars = compat_parse_qs(self._html_search_regex(
|
||||
r'<param name="flashvars" value="([^"]+)"', webpage, 'flashvars'))
|
||||
|
||||
page_id = flashvars['trackerClipId'][0]
|
||||
video_url = flashvars['dslSrc'][0]
|
||||
title = flashvars['trackerClipTitle'][0]
|
||||
thumbnail = flashvars['startPicture'][0] if 'startPicture' in flashvars else None
|
||||
is_live = flashvars.get('isLive', ['0'])[0] == '1'
|
||||
|
||||
if is_live:
|
||||
info_dict.update({
|
||||
'title': self._live_title(info_dict['title']),
|
||||
'upload_date': None,
|
||||
title = self._live_title(title)
|
||||
|
||||
if 'trackerClipAirTime' in flashvars:
|
||||
upload_date = flashvars['trackerClipAirTime'][0]
|
||||
else:
|
||||
upload_date = self._html_search_meta(
|
||||
'DC.Date', webpage, 'upload date')
|
||||
|
||||
if upload_date:
|
||||
upload_date = unified_strdate(upload_date)
|
||||
|
||||
formats = []
|
||||
preference = qualities(['S', 'M', 'L', 'XL'])
|
||||
|
||||
if video_url.endswith('.f4m'):
|
||||
formats.extend(self._extract_f4m_formats(
|
||||
video_url + '?hdcore=3.2.0&plugin=aasp-3.2.0.77.18', page_id,
|
||||
f4m_id='hds', fatal=False))
|
||||
elif video_url.endswith('.smil'):
|
||||
formats.extend(self._extract_smil_formats(
|
||||
video_url, page_id, False, {
|
||||
'hdcore': '3.3.0',
|
||||
'plugin': 'aasp-3.3.0.99.43',
|
||||
}))
|
||||
else:
|
||||
formats.append({
|
||||
'url': video_url,
|
||||
'http_headers': {
|
||||
'User-Agent': 'mobile',
|
||||
},
|
||||
})
|
||||
elif 'upload_date' not in info_dict:
|
||||
info_dict['upload_date'] = unified_strdate(self._html_search_meta('DC.Date', webpage, 'upload date'))
|
||||
|
||||
info_dict.update({
|
||||
'description': self._html_search_meta('Description', webpage),
|
||||
'is_live': is_live,
|
||||
})
|
||||
m3u8_url = self._search_regex(
|
||||
r'rel="adaptiv"[^>]+href="([^"]+)"',
|
||||
webpage, 'm3u8 url', default=None)
|
||||
if m3u8_url:
|
||||
formats.extend(self._extract_m3u8_formats(
|
||||
m3u8_url, page_id, 'mp4', 'm3u8_native',
|
||||
m3u8_id='hls', fatal=False))
|
||||
|
||||
return info_dict
|
||||
direct_urls = re.findall(
|
||||
r'rel="web(S|M|L|XL)"[^>]+href="([^"]+)"', webpage)
|
||||
if direct_urls:
|
||||
for quality, video_url in direct_urls:
|
||||
formats.append({
|
||||
'url': video_url,
|
||||
'preference': preference(quality),
|
||||
'http_headers': {
|
||||
'User-Agent': 'mobile',
|
||||
},
|
||||
})
|
||||
|
||||
self._sort_formats(formats)
|
||||
|
||||
description = self._html_search_meta('Description', webpage, 'description')
|
||||
|
||||
return {
|
||||
'id': page_id,
|
||||
'formats': formats,
|
||||
'title': title,
|
||||
'description': description,
|
||||
'thumbnail': thumbnail,
|
||||
'upload_date': upload_date,
|
||||
'is_live': is_live
|
||||
}
|
||||
|
||||
|
||||
class WDRMobileIE(InfoExtractor):
|
||||
@@ -262,3 +241,81 @@ class WDRMobileIE(InfoExtractor):
|
||||
'User-Agent': 'mobile',
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class WDRMausIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?wdrmaus\.de/(?:[^/]+/){,2}(?P<id>[^/?#]+)(?:/index\.php5|(?<!index)\.php5|/(?:$|[?#]))'
|
||||
IE_DESC = 'Sendung mit der Maus'
|
||||
_TESTS = [{
|
||||
'url': 'http://www.wdrmaus.de/aktuelle-sendung/index.php5',
|
||||
'info_dict': {
|
||||
'id': 'aktuelle-sendung',
|
||||
'ext': 'mp4',
|
||||
'thumbnail': 're:^http://.+\.jpg',
|
||||
'upload_date': 're:^[0-9]{8}$',
|
||||
'title': 're:^[0-9.]{10} - Aktuelle Sendung$',
|
||||
}
|
||||
}, {
|
||||
'url': 'http://www.wdrmaus.de/sachgeschichten/sachgeschichten/40_jahre_maus.php5',
|
||||
'md5': '3b1227ca3ed28d73ec5737c65743b2a3',
|
||||
'info_dict': {
|
||||
'id': '40_jahre_maus',
|
||||
'ext': 'mp4',
|
||||
'thumbnail': 're:^http://.+\.jpg',
|
||||
'upload_date': '20131007',
|
||||
'title': '12.03.2011 - 40 Jahre Maus',
|
||||
}
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
param_code = self._html_search_regex(
|
||||
r'<a href="\?startVideo=1&([^"]+)"', webpage, 'parameters')
|
||||
|
||||
title_date = self._search_regex(
|
||||
r'<div class="sendedatum"><p>Sendedatum:\s*([0-9\.]+)</p>',
|
||||
webpage, 'air date')
|
||||
title_str = self._html_search_regex(
|
||||
r'<h1>(.*?)</h1>', webpage, 'title')
|
||||
title = '%s - %s' % (title_date, title_str)
|
||||
upload_date = unified_strdate(
|
||||
self._html_search_meta('dc.date', webpage))
|
||||
|
||||
fields = compat_parse_qs(param_code)
|
||||
video_url = fields['firstVideo'][0]
|
||||
thumbnail = compat_urlparse.urljoin(url, fields['startPicture'][0])
|
||||
|
||||
formats = [{
|
||||
'format_id': 'rtmp',
|
||||
'url': video_url,
|
||||
}]
|
||||
|
||||
jscode = self._download_webpage(
|
||||
'http://www.wdrmaus.de/codebase/js/extended-medien.min.js',
|
||||
video_id, fatal=False,
|
||||
note='Downloading URL translation table',
|
||||
errnote='Could not download URL translation table')
|
||||
if jscode:
|
||||
for m in re.finditer(
|
||||
r"stream:\s*'dslSrc=(?P<stream>[^']+)',\s*download:\s*'(?P<dl>[^']+)'\s*\}",
|
||||
jscode):
|
||||
if video_url.startswith(m.group('stream')):
|
||||
http_url = video_url.replace(
|
||||
m.group('stream'), m.group('dl'))
|
||||
formats.append({
|
||||
'format_id': 'http',
|
||||
'url': http_url,
|
||||
})
|
||||
break
|
||||
|
||||
self._sort_formats(formats)
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'title': title,
|
||||
'formats': formats,
|
||||
'thumbnail': thumbnail,
|
||||
'upload_date': upload_date,
|
||||
}
|
||||
|
||||
49
youtube_dl/extractor/weibo.py
Normal file
49
youtube_dl/extractor/weibo.py
Normal file
@@ -0,0 +1,49 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
|
||||
|
||||
class WeiboIE(InfoExtractor):
|
||||
"""
|
||||
The videos in Weibo come from different sites, this IE just finds the link
|
||||
to the external video and returns it.
|
||||
"""
|
||||
_VALID_URL = r'https?://video\.weibo\.com/v/weishipin/t_(?P<id>.+?)\.htm'
|
||||
|
||||
_TEST = {
|
||||
'url': 'http://video.weibo.com/v/weishipin/t_zjUw2kZ.htm',
|
||||
'info_dict': {
|
||||
'id': '98322879',
|
||||
'ext': 'flv',
|
||||
'title': '魔声耳机最新广告“All Eyes On Us”',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
'add_ie': ['Sina'],
|
||||
}
|
||||
|
||||
# Additional example videos from different sites
|
||||
# Youku: http://video.weibo.com/v/weishipin/t_zQGDWQ8.htm
|
||||
# 56.com: http://video.weibo.com/v/weishipin/t_zQ44HxN.htm
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url, flags=re.VERBOSE)
|
||||
video_id = mobj.group('id')
|
||||
info_url = 'http://video.weibo.com/?s=v&a=play_list&format=json&mix_video_id=t_%s' % video_id
|
||||
info = self._download_json(info_url, video_id)
|
||||
|
||||
videos_urls = map(lambda v: v['play_page_url'], info['result']['data'])
|
||||
# Prefer sina video since they have thumbnails
|
||||
videos_urls = sorted(videos_urls, key=lambda u: 'video.sina.com' in u)
|
||||
player_url = videos_urls[-1]
|
||||
m_sina = re.match(r'https?://video\.sina\.com\.cn/v/b/(\d+)-\d+\.html',
|
||||
player_url)
|
||||
if m_sina is not None:
|
||||
self.to_screen('Sina video detected')
|
||||
sina_id = m_sina.group(1)
|
||||
player_url = 'http://you.video.sina.com.cn/swf/quotePlayer.swf?vid=%s' % sina_id
|
||||
return self.url_result(player_url)
|
||||
@@ -1,33 +1,29 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
from .common import InfoExtractor
|
||||
from .youtube import YoutubeIE
|
||||
from .jwplatform import JWPlatformBaseIE
|
||||
|
||||
|
||||
class WimpIE(JWPlatformBaseIE):
|
||||
class WimpIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?wimp\.com/(?P<id>[^/]+)'
|
||||
_TESTS = [{
|
||||
'url': 'http://www.wimp.com/maru-is-exhausted/',
|
||||
'url': 'http://www.wimp.com/maruexhausted/',
|
||||
'md5': 'ee21217ffd66d058e8b16be340b74883',
|
||||
'info_dict': {
|
||||
'id': 'maru-is-exhausted',
|
||||
'id': 'maruexhausted',
|
||||
'ext': 'mp4',
|
||||
'title': 'Maru is exhausted.',
|
||||
'description': 'md5:57e099e857c0a4ea312542b684a869b8',
|
||||
}
|
||||
}, {
|
||||
'url': 'http://www.wimp.com/clowncar/',
|
||||
'md5': '5c31ad862a90dc5b1f023956faec13fe',
|
||||
'md5': '4e2986c793694b55b37cf92521d12bb4',
|
||||
'info_dict': {
|
||||
'id': 'cG4CEr2aiSg',
|
||||
'id': 'clowncar',
|
||||
'ext': 'webm',
|
||||
'title': 'Basset hound clown car...incredible!',
|
||||
'description': '5 of my Bassets crawled in this dog loo! www.bellinghambassets.com\n\nFor licensing/usage please contact: licensing(at)jukinmediadotcom',
|
||||
'upload_date': '20140303',
|
||||
'uploader': 'Gretchen Hoey',
|
||||
'uploader_id': 'gretchenandjeff1',
|
||||
'title': 'It\'s like a clown car.',
|
||||
'description': 'md5:0e56db1370a6e49c5c1d19124c0d2fb2',
|
||||
},
|
||||
'add_ie': ['Youtube'],
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
@@ -45,13 +41,14 @@ class WimpIE(JWPlatformBaseIE):
|
||||
'ie_key': YoutubeIE.ie_key(),
|
||||
}
|
||||
|
||||
info_dict = self._extract_jwplayer_data(
|
||||
webpage, video_id, require_title=False)
|
||||
video_url = self._search_regex(
|
||||
r'<video[^>]+>\s*<source[^>]+src=(["\'])(?P<url>.+?)\1',
|
||||
webpage, 'video URL', group='url')
|
||||
|
||||
info_dict.update({
|
||||
return {
|
||||
'id': video_id,
|
||||
'url': video_url,
|
||||
'title': self._og_search_title(webpage),
|
||||
'thumbnail': self._og_search_thumbnail(webpage),
|
||||
'description': self._og_search_description(webpage),
|
||||
})
|
||||
|
||||
return info_dict
|
||||
}
|
||||
|
||||
@@ -5,10 +5,8 @@ import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
int_or_none,
|
||||
qualities,
|
||||
remove_start,
|
||||
)
|
||||
|
||||
|
||||
@@ -28,17 +26,16 @@ class WrzutaIE(InfoExtractor):
|
||||
'uploader_id': 'laboratoriumdextera',
|
||||
'description': 'md5:7fb5ef3c21c5893375fda51d9b15d9cd',
|
||||
},
|
||||
'skip': 'Redirected to wrzuta.pl',
|
||||
}, {
|
||||
'url': 'http://vexling.wrzuta.pl/audio/01xBFabGXu6/james_horner_-_into_the_na_39_vi_world_bonus',
|
||||
'md5': 'f80564fb5a2ec6ec59705ae2bf2ba56d',
|
||||
'url': 'http://jolka85.wrzuta.pl/audio/063jOPX5ue2/liber_natalia_szroeder_-_teraz_ty',
|
||||
'md5': 'bc78077859bea7bcfe4295d7d7fc9025',
|
||||
'info_dict': {
|
||||
'id': '01xBFabGXu6',
|
||||
'ext': 'mp3',
|
||||
'title': 'James Horner - Into The Na\'vi World [Bonus]',
|
||||
'description': 'md5:30a70718b2cd9df3120fce4445b0263b',
|
||||
'duration': 95,
|
||||
'uploader_id': 'vexling',
|
||||
'id': '063jOPX5ue2',
|
||||
'ext': 'ogg',
|
||||
'title': 'Liber & Natalia Szroeder - Teraz Ty',
|
||||
'duration': 203,
|
||||
'uploader_id': 'jolka85',
|
||||
'description': 'md5:2d2b6340f9188c8c4cd891580e481096',
|
||||
},
|
||||
}]
|
||||
|
||||
@@ -48,10 +45,7 @@ class WrzutaIE(InfoExtractor):
|
||||
typ = mobj.group('typ')
|
||||
uploader = mobj.group('uploader')
|
||||
|
||||
webpage, urlh = self._download_webpage_handle(url, video_id)
|
||||
|
||||
if urlh.geturl() == 'http://www.wrzuta.pl/':
|
||||
raise ExtractorError('Video removed', expected=True)
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
quality = qualities(['SD', 'MQ', 'HQ', 'HD'])
|
||||
|
||||
@@ -86,73 +80,3 @@ class WrzutaIE(InfoExtractor):
|
||||
'description': self._og_search_description(webpage),
|
||||
'age_limit': embedpage.get('minimalAge', 0),
|
||||
}
|
||||
|
||||
|
||||
class WrzutaPlaylistIE(InfoExtractor):
|
||||
"""
|
||||
this class covers extraction of wrzuta playlist entries
|
||||
the extraction process bases on following steps:
|
||||
* collect information of playlist size
|
||||
* download all entries provided on
|
||||
the playlist webpage (the playlist is split
|
||||
on two pages: first directly reached from webpage
|
||||
second: downloaded on demand by ajax call and rendered
|
||||
using the ajax call response)
|
||||
* in case size of extracted entries not reached total number of entries
|
||||
use the ajax call to collect the remaining entries
|
||||
"""
|
||||
|
||||
IE_NAME = 'wrzuta.pl:playlist'
|
||||
_VALID_URL = r'https?://(?P<uploader>[0-9a-zA-Z]+)\.wrzuta\.pl/playlista/(?P<id>[0-9a-zA-Z]+)'
|
||||
_TESTS = [{
|
||||
'url': 'http://miromak71.wrzuta.pl/playlista/7XfO4vE84iR/moja_muza',
|
||||
'playlist_mincount': 14,
|
||||
'info_dict': {
|
||||
'id': '7XfO4vE84iR',
|
||||
'title': 'Moja muza',
|
||||
},
|
||||
}, {
|
||||
'url': 'http://heroesf70.wrzuta.pl/playlista/6Nj3wQHx756/lipiec_-_lato_2015_muzyka_swiata',
|
||||
'playlist_mincount': 144,
|
||||
'info_dict': {
|
||||
'id': '6Nj3wQHx756',
|
||||
'title': 'Lipiec - Lato 2015 Muzyka Świata',
|
||||
},
|
||||
}, {
|
||||
'url': 'http://miromak71.wrzuta.pl/playlista/7XfO4vE84iR',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
playlist_id = mobj.group('id')
|
||||
uploader = mobj.group('uploader')
|
||||
|
||||
webpage = self._download_webpage(url, playlist_id)
|
||||
|
||||
playlist_size = int_or_none(self._html_search_regex(
|
||||
(r'<div[^>]+class=["\']playlist-counter["\'][^>]*>\d+/(\d+)',
|
||||
r'<div[^>]+class=["\']all-counter["\'][^>]*>(.+?)</div>'),
|
||||
webpage, 'playlist size', default=None))
|
||||
|
||||
playlist_title = remove_start(
|
||||
self._og_search_title(webpage), 'Playlista: ')
|
||||
|
||||
entries = []
|
||||
if playlist_size:
|
||||
entries = [
|
||||
self.url_result(entry_url)
|
||||
for _, entry_url in re.findall(
|
||||
r'<a[^>]+href=(["\'])(http.+?)\1[^>]+class=["\']playlist-file-page',
|
||||
webpage)]
|
||||
if playlist_size > len(entries):
|
||||
playlist_content = self._download_json(
|
||||
'http://%s.wrzuta.pl/xhr/get_playlist_offset/%s' % (uploader, playlist_id),
|
||||
playlist_id,
|
||||
'Downloading playlist JSON',
|
||||
'Unable to download playlist JSON')
|
||||
entries.extend([
|
||||
self.url_result(entry['filelink'])
|
||||
for entry in playlist_content.get('files', []) if entry.get('filelink')])
|
||||
|
||||
return self.playlist_result(entries, playlist_id, playlist_title)
|
||||
|
||||
@@ -5,10 +5,8 @@ import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
decode_packed_codes,
|
||||
ExtractorError,
|
||||
int_or_none,
|
||||
NO_DEFAULT,
|
||||
sanitized_Request,
|
||||
urlencode_postdata,
|
||||
)
|
||||
@@ -25,24 +23,20 @@ class XFileShareIE(InfoExtractor):
|
||||
('thevideobee.to', 'TheVideoBee'),
|
||||
('vidto.me', 'Vidto'),
|
||||
('streamin.to', 'Streamin.To'),
|
||||
('xvidstage.com', 'XVIDSTAGE'),
|
||||
)
|
||||
|
||||
IE_DESC = 'XFileShare based sites: %s' % ', '.join(list(zip(*_SITES))[1])
|
||||
_VALID_URL = (r'https?://(?P<host>(?:www\.)?(?:%s))/(?:embed-)?(?P<id>[0-9a-zA-Z]+)'
|
||||
% '|'.join(re.escape(site) for site in list(zip(*_SITES))[0]))
|
||||
|
||||
_FILE_NOT_FOUND_REGEXES = (
|
||||
r'>(?:404 - )?File Not Found<',
|
||||
r'>The file was removed by administrator<',
|
||||
)
|
||||
_FILE_NOT_FOUND_REGEX = r'>(?:404 - )?File Not Found<'
|
||||
|
||||
_TESTS = [{
|
||||
'url': 'http://gorillavid.in/06y9juieqpmi',
|
||||
'md5': '5ae4a3580620380619678ee4875893ba',
|
||||
'info_dict': {
|
||||
'id': '06y9juieqpmi',
|
||||
'ext': 'mp4',
|
||||
'ext': 'flv',
|
||||
'title': 'Rebecca Black My Moment Official Music Video Reaction-6GK87Rc8bzQ',
|
||||
'thumbnail': 're:http://.*\.jpg',
|
||||
},
|
||||
@@ -68,8 +62,7 @@ class XFileShareIE(InfoExtractor):
|
||||
'ext': 'mp4',
|
||||
'title': 'youtube-dl test video \'äBaW_jenozKc.mp4.mp4',
|
||||
'thumbnail': 're:http://.*\.jpg',
|
||||
},
|
||||
'skip': 'Video removed',
|
||||
}
|
||||
}, {
|
||||
'url': 'http://vidto.me/ku5glz52nqe1.html',
|
||||
'info_dict': {
|
||||
@@ -84,17 +77,6 @@ class XFileShareIE(InfoExtractor):
|
||||
'ext': 'mp4',
|
||||
'title': 'Big Buck Bunny trailer',
|
||||
},
|
||||
}, {
|
||||
'url': 'http://xvidstage.com/e0qcnl03co6z',
|
||||
'info_dict': {
|
||||
'id': 'e0qcnl03co6z',
|
||||
'ext': 'mp4',
|
||||
'title': 'Chucky Prank 2015.mp4',
|
||||
},
|
||||
}, {
|
||||
# removed by administrator
|
||||
'url': 'http://xvidstage.com/amfy7atlkx25',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
@@ -104,7 +86,7 @@ class XFileShareIE(InfoExtractor):
|
||||
url = 'http://%s/%s' % (mobj.group('host'), video_id)
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
if any(re.search(p, webpage) for p in self._FILE_NOT_FOUND_REGEXES):
|
||||
if re.search(self._FILE_NOT_FOUND_REGEX, webpage) is not None:
|
||||
raise ExtractorError('Video %s does not exist' % video_id, expected=True)
|
||||
|
||||
fields = self._hidden_inputs(webpage)
|
||||
@@ -130,23 +112,10 @@ class XFileShareIE(InfoExtractor):
|
||||
r'>Watch (.+) ',
|
||||
r'<h2 class="video-page-head">([^<]+)</h2>'],
|
||||
webpage, 'title', default=None) or self._og_search_title(webpage)).strip()
|
||||
|
||||
def extract_video_url(default=NO_DEFAULT):
|
||||
return self._search_regex(
|
||||
(r'file\s*:\s*(["\'])(?P<url>http.+?)\1,',
|
||||
r'file_link\s*=\s*(["\'])(?P<url>http.+?)\1',
|
||||
r'addVariable\((\\?["\'])file\1\s*,\s*(\\?["\'])(?P<url>http.+?)\2\)',
|
||||
r'<embed[^>]+src=(["\'])(?P<url>http.+?)\1'),
|
||||
webpage, 'file url', default=default, group='url')
|
||||
|
||||
video_url = extract_video_url(default=None)
|
||||
|
||||
if not video_url:
|
||||
webpage = decode_packed_codes(self._search_regex(
|
||||
r"(}\('(.+)',(\d+),(\d+),'[^']*\b(?:file|embed)\b[^']*'\.split\('\|'\))",
|
||||
webpage, 'packed code'))
|
||||
video_url = extract_video_url()
|
||||
|
||||
video_url = self._search_regex(
|
||||
[r'file\s*:\s*["\'](http[^"\']+)["\'],',
|
||||
r'file_link\s*=\s*\'(https?:\/\/[0-9a-zA-z.\/\-_]+)'],
|
||||
webpage, 'file url')
|
||||
thumbnail = self._search_regex(
|
||||
r'image\s*:\s*["\'](http[^"\']+)["\'],', webpage, 'thumbnail', default=None)
|
||||
|
||||
|
||||
@@ -66,7 +66,6 @@ class XuiteIE(InfoExtractor):
|
||||
'uploader_id': '242127761',
|
||||
'categories': ['電玩動漫'],
|
||||
},
|
||||
'skip': 'Video removed',
|
||||
}, {
|
||||
'url': 'http://vlog.xuite.net/play/S1dDUjdyLTMyOTc3NjcuZmx2/%E5%AD%AB%E7%87%95%E5%A7%BF-%E7%9C%BC%E6%B7%9A%E6%88%90%E8%A9%A9',
|
||||
'only_matching': True,
|
||||
|
||||
@@ -343,7 +343,7 @@ class YahooIE(InfoExtractor):
|
||||
webpage, 'region', fatal=False, default='US')
|
||||
data = compat_urllib_parse_urlencode({
|
||||
'protocol': 'http',
|
||||
'region': region.upper(),
|
||||
'region': region,
|
||||
})
|
||||
query_url = (
|
||||
'https://video.media.yql.yahoo.com/v1/video/sapi/streams/'
|
||||
|
||||
@@ -2,9 +2,7 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import base64
|
||||
import itertools
|
||||
import random
|
||||
import re
|
||||
import string
|
||||
import time
|
||||
|
||||
@@ -15,7 +13,6 @@ from ..compat import (
|
||||
)
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
get_element_by_attribute,
|
||||
sanitized_Request,
|
||||
)
|
||||
|
||||
@@ -288,52 +285,3 @@ class YoukuIE(InfoExtractor):
|
||||
'title': title,
|
||||
'entries': entries,
|
||||
}
|
||||
|
||||
|
||||
class YoukuShowIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?youku\.com/show_page/id_(?P<id>[0-9a-z]+)\.html'
|
||||
IE_NAME = 'youku:show'
|
||||
|
||||
_TEST = {
|
||||
'url': 'http://www.youku.com/show_page/id_zc7c670be07ff11e48b3f.html',
|
||||
'info_dict': {
|
||||
'id': 'zc7c670be07ff11e48b3f',
|
||||
'title': '花千骨 未删减版',
|
||||
'description': 'md5:578d4f2145ae3f9128d9d4d863312910',
|
||||
},
|
||||
'playlist_count': 50,
|
||||
}
|
||||
|
||||
_PAGE_SIZE = 40
|
||||
|
||||
def _find_videos_in_page(self, webpage):
|
||||
videos = re.findall(
|
||||
r'<li><a[^>]+href="(?P<url>https?://v\.youku\.com/[^"]+)"[^>]+title="(?P<title>[^"]+)"', webpage)
|
||||
return [
|
||||
self.url_result(video_url, YoukuIE.ie_key(), title)
|
||||
for video_url, title in videos]
|
||||
|
||||
def _real_extract(self, url):
|
||||
show_id = self._match_id(url)
|
||||
webpage = self._download_webpage(url, show_id)
|
||||
|
||||
entries = self._find_videos_in_page(webpage)
|
||||
|
||||
playlist_title = self._html_search_regex(
|
||||
r'<span[^>]+class="name">([^<]+)</span>', webpage, 'playlist title', fatal=False)
|
||||
detail_div = get_element_by_attribute('class', 'detail', webpage) or ''
|
||||
playlist_description = self._html_search_regex(
|
||||
r'<span[^>]+style="display:none"[^>]*>([^<]+)</span>',
|
||||
detail_div, 'playlist description', fatal=False)
|
||||
|
||||
for idx in itertools.count(1):
|
||||
episodes_page = self._download_webpage(
|
||||
'http://www.youku.com/show_episode/id_%s.html' % show_id,
|
||||
show_id, query={'divid': 'reload_%d' % (idx * self._PAGE_SIZE + 1)},
|
||||
note='Downloading episodes page %d' % idx)
|
||||
new_entries = self._find_videos_in_page(episodes_page)
|
||||
entries.extend(new_entries)
|
||||
if len(new_entries) < self._PAGE_SIZE:
|
||||
break
|
||||
|
||||
return self.playlist_result(entries, show_id, playlist_title, playlist_description)
|
||||
|
||||
@@ -17,7 +17,7 @@ class YouPornIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?youporn\.com/watch/(?P<id>\d+)/(?P<display_id>[^/?#&]+)'
|
||||
_TESTS = [{
|
||||
'url': 'http://www.youporn.com/watch/505835/sex-ed-is-it-safe-to-masturbate-daily/',
|
||||
'md5': '3744d24c50438cf5b6f6d59feb5055c2',
|
||||
'md5': '71ec5fcfddacf80f495efa8b6a8d9a89',
|
||||
'info_dict': {
|
||||
'id': '505835',
|
||||
'display_id': 'sex-ed-is-it-safe-to-masturbate-daily',
|
||||
@@ -121,21 +121,21 @@ class YouPornIE(InfoExtractor):
|
||||
webpage, 'thumbnail', fatal=False, group='thumbnail')
|
||||
|
||||
uploader = self._html_search_regex(
|
||||
r'(?s)<div[^>]+class=["\']submitByLink["\'][^>]*>(.+?)</div>',
|
||||
r'(?s)<div[^>]+class=["\']videoInfoBy(?:\s+[^"\']+)?["\'][^>]*>\s*By:\s*</div>(.+?)</(?:a|div)>',
|
||||
webpage, 'uploader', fatal=False)
|
||||
upload_date = unified_strdate(self._html_search_regex(
|
||||
r'(?s)<div[^>]+class=["\']videoInfo(?:Date|Time)["\'][^>]*>(.+?)</div>',
|
||||
r'(?s)<div[^>]+class=["\']videoInfoTime["\'][^>]*>(.+?)</div>',
|
||||
webpage, 'upload date', fatal=False))
|
||||
|
||||
age_limit = self._rta_search(webpage)
|
||||
|
||||
average_rating = int_or_none(self._search_regex(
|
||||
r'<div[^>]+class=["\']videoRatingPercentage["\'][^>]*>(\d+)%</div>',
|
||||
r'<div[^>]+class=["\']videoInfoRating["\'][^>]*>\s*<div[^>]+class=["\']videoRatingPercentage["\'][^>]*>(\d+)%</div>',
|
||||
webpage, 'average rating', fatal=False))
|
||||
|
||||
view_count = str_to_int(self._search_regex(
|
||||
r'(?s)<div[^>]+class=(["\']).*?\bvideoInfoViews\b.*?\1[^>]*>.*?(?P<count>[\d,.]+)<',
|
||||
webpage, 'view count', fatal=False, group='count'))
|
||||
r'(?s)<div[^>]+class=["\']videoInfoViews["\'][^>]*>.*?([\d,.]+)\s*</div>',
|
||||
webpage, 'view count', fatal=False))
|
||||
comment_count = str_to_int(self._search_regex(
|
||||
r'>All [Cc]omments? \(([\d,.]+)\)',
|
||||
webpage, 'comment count', fatal=False))
|
||||
|
||||
@@ -344,8 +344,6 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
|
||||
'139': {'ext': 'm4a', 'format_note': 'DASH audio', 'acodec': 'aac', 'abr': 48, 'preference': -50, 'container': 'm4a_dash'},
|
||||
'140': {'ext': 'm4a', 'format_note': 'DASH audio', 'acodec': 'aac', 'abr': 128, 'preference': -50, 'container': 'm4a_dash'},
|
||||
'141': {'ext': 'm4a', 'format_note': 'DASH audio', 'acodec': 'aac', 'abr': 256, 'preference': -50, 'container': 'm4a_dash'},
|
||||
'256': {'ext': 'm4a', 'format_note': 'DASH audio', 'acodec': 'aac', 'preference': -50, 'container': 'm4a_dash'},
|
||||
'258': {'ext': 'm4a', 'format_note': 'DASH audio', 'acodec': 'aac', 'preference': -50, 'container': 'm4a_dash'},
|
||||
|
||||
# Dash webm
|
||||
'167': {'ext': 'webm', 'height': 360, 'width': 640, 'format_note': 'DASH video', 'container': 'webm', 'vcodec': 'vp8', 'preference': -40},
|
||||
@@ -1988,7 +1986,7 @@ class YoutubeChannelIE(YoutubePlaylistBaseInfoExtractor):
|
||||
|
||||
class YoutubeUserIE(YoutubeChannelIE):
|
||||
IE_DESC = 'YouTube.com user videos (URL or "ytuser" keyword)'
|
||||
_VALID_URL = r'(?:(?:https?://(?:\w+\.)?youtube\.com/(?:user/|c/)?(?!(?:attribution_link|watch|results)(?:$|[^a-z_A-Z0-9-])))|ytuser:)(?!feed/)(?P<id>[A-Za-z0-9_-]+)'
|
||||
_VALID_URL = r'(?:(?:https?://(?:\w+\.)?youtube\.com/(?:user/)?(?!(?:attribution_link|watch|results)(?:$|[^a-z_A-Z0-9-])))|ytuser:)(?!feed/)(?P<id>[A-Za-z0-9_-]+)'
|
||||
_TEMPLATE_URL = 'https://www.youtube.com/user/%s/videos'
|
||||
IE_NAME = 'youtube:user'
|
||||
|
||||
@@ -2001,9 +1999,6 @@ class YoutubeUserIE(YoutubeChannelIE):
|
||||
}, {
|
||||
'url': 'ytuser:phihag',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://www.youtube.com/c/gametrailers',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
@classmethod
|
||||
|
||||
@@ -131,8 +131,9 @@ class JSInterpreter(object):
|
||||
if variable in local_vars:
|
||||
obj = local_vars[variable]
|
||||
else:
|
||||
obj = self._objects.setdefault(
|
||||
variable, self.extract_object(variable))
|
||||
if variable not in self._objects:
|
||||
self._objects[variable] = self.extract_object(variable)
|
||||
obj = self._objects[variable]
|
||||
|
||||
if arg_str is None:
|
||||
# Member access
|
||||
@@ -203,7 +204,8 @@ class JSInterpreter(object):
|
||||
argvals = tuple([
|
||||
int(v) if v.isdigit() else local_vars[v]
|
||||
for v in m.group('args').split(',')])
|
||||
self._functions.setdefault(fname, self.extract_function(fname))
|
||||
if fname not in self._functions:
|
||||
self._functions[fname] = self.extract_function(fname)
|
||||
return self._functions[fname](argvals)
|
||||
|
||||
raise ExtractorError('Unsupported JS expression %r' % expr)
|
||||
|
||||
@@ -668,7 +668,7 @@ def parseOpts(overrideArguments=None):
|
||||
action='store_true', dest='writeannotations', default=False,
|
||||
help='Write video annotations to a .annotations.xml file')
|
||||
filesystem.add_option(
|
||||
'--load-info-json', '--load-info',
|
||||
'--load-info',
|
||||
dest='load_info_filename', metavar='FILE',
|
||||
help='JSON file containing the video information (created with the "--write-info-json" option)')
|
||||
filesystem.add_option(
|
||||
|
||||
@@ -39,7 +39,6 @@ from .compat import (
|
||||
compat_chr,
|
||||
compat_etree_fromstring,
|
||||
compat_html_entities,
|
||||
compat_html_entities_html5,
|
||||
compat_http_client,
|
||||
compat_kwargs,
|
||||
compat_parse_qs,
|
||||
@@ -76,7 +75,7 @@ def register_socks_protocols():
|
||||
compiled_regex_type = type(re.compile(''))
|
||||
|
||||
std_headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)',
|
||||
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/44.0 (Chrome)',
|
||||
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
|
||||
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
|
||||
'Accept-Encoding': 'gzip, deflate',
|
||||
@@ -457,19 +456,12 @@ def orderedSet(iterable):
|
||||
return res
|
||||
|
||||
|
||||
def _htmlentity_transform(entity_with_semicolon):
|
||||
def _htmlentity_transform(entity):
|
||||
"""Transforms an HTML entity to a character."""
|
||||
entity = entity_with_semicolon[:-1]
|
||||
|
||||
# Known non-numeric HTML entity
|
||||
if entity in compat_html_entities.name2codepoint:
|
||||
return compat_chr(compat_html_entities.name2codepoint[entity])
|
||||
|
||||
# TODO: HTML5 allows entities without a semicolon. For example,
|
||||
# 'Éric' should be decoded as 'Éric'.
|
||||
if entity_with_semicolon in compat_html_entities_html5:
|
||||
return compat_html_entities_html5[entity_with_semicolon]
|
||||
|
||||
mobj = re.match(r'#(x[0-9a-fA-F]+|[0-9]+)', entity)
|
||||
if mobj is not None:
|
||||
numstr = mobj.group(1)
|
||||
@@ -494,7 +486,7 @@ def unescapeHTML(s):
|
||||
assert type(s) == compat_str
|
||||
|
||||
return re.sub(
|
||||
r'&([^;]+;)', lambda m: _htmlentity_transform(m.group(1)), s)
|
||||
r'&([^;]+);', lambda m: _htmlentity_transform(m.group(1)), s)
|
||||
|
||||
|
||||
def get_subprocess_encoding():
|
||||
@@ -1901,16 +1893,6 @@ def dict_get(d, key_or_keys, default=None, skip_false_values=True):
|
||||
return d.get(key_or_keys, default)
|
||||
|
||||
|
||||
def try_get(src, getter, expected_type=None):
|
||||
try:
|
||||
v = getter(src)
|
||||
except (AttributeError, KeyError, TypeError, IndexError):
|
||||
pass
|
||||
else:
|
||||
if expected_type is None or isinstance(v, expected_type):
|
||||
return v
|
||||
|
||||
|
||||
def encode_compat_str(string, encoding=preferredencoding(), errors='strict'):
|
||||
return string if isinstance(string, compat_str) else compat_str(string, encoding, errors)
|
||||
|
||||
@@ -1970,7 +1952,7 @@ def js_to_json(code):
|
||||
'(?:[^'\\]*(?:\\\\|\\['"nurtbfx/\n]))*[^'\\]*'|
|
||||
/\*.*?\*/|,(?=\s*[\]}])|
|
||||
[a-zA-Z_][.a-zA-Z_0-9]*|
|
||||
\b(?:0[xX][0-9a-fA-F]+|0+[0-7]+)(?:\s*:)?|
|
||||
(?:0[xX][0-9a-fA-F]+|0+[0-7]+)(?:\s*:)?|
|
||||
[0-9]+(?=\s*:)
|
||||
''', fix_kv, code)
|
||||
|
||||
@@ -2038,9 +2020,6 @@ def mimetype2ext(mt):
|
||||
|
||||
ext = {
|
||||
'audio/mp4': 'm4a',
|
||||
# Per RFC 3003, audio/mpeg can be .mp1, .mp2 or .mp3. Here use .mp3 as
|
||||
# it's the most popular one
|
||||
'audio/mpeg': 'mp3',
|
||||
}.get(mt)
|
||||
if ext is not None:
|
||||
return ext
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
__version__ = '2016.06.19.1'
|
||||
__version__ = '2016.06.04'
|
||||
|
||||
Reference in New Issue
Block a user