Apr 29 2009

Update 6th May 2009: You might want to read this first…

A milestone has been reached today with the addition of Channel 5’s Demand Five service to the list of DRM-free services that get_iplayer can index, search and record. The following are now supported:

Supported platforms are: Linux, MacOSX, Windows and pretty much any modern OS that can run perl. See the download and installation pages.

To download and search programmes use:

get_iplayer –type=TYPE <other options>

Where TYPE is either ‘tv’ (BBC TV), ‘itv’, ‘ch4’, ‘five’, ‘radio’ (BBC Radio), or ‘podcast’ (BBC).

For full instructions see the documentation page

Now I’m off for a beer….

54 Comments

  • On April 30th, 2009, linuxcentre said:

    Your rtmpdump is too old – needs to be v.1.5 or better.
    If you are on MacOSX then you need to compile rtmpdump v1.5 from source. I don’t have access to a mac so if anyone can help here please email the binary to iplayer2 (at sign) linuxcentre.net

    • On April 30th, 2009, TG said:

      Am on a Mac. Working on compiling rtmpdump from source, however it throws up some errors which I believe are related to needing to update OpenSSL. Linux is just so much easier sometimes…

      I’ll email you the file if I can get it compiled

      • On April 30th, 2009, linuxcentre said:

        The README file in the tarball for rtmpdump has some pointers generally on compiling. Do you by any chance need the development package for openssl (i.e. the headers etc) before you can compile against this lib?

        • On April 30th, 2009, TG said:

          The headers etc. are all in place. I’m pretty sure the issue is that the version of OpenSSL included with OS X (10.5.6 in my case) is compiled without SHA256 support.

          The only nice way around this I can think of is to statically link a OpenSSL library (compiled with SHA256 support) to the rtmpdump binary, so this can be distributed without the need for messing around with the system files – something which really shouldn’t be necessary.

          I’ll have a go – but if you have any pointers on how to build it like this that’d be great 🙂

          • On April 30th, 2009, TG said:

            OK, managed to build it, using above method 🙂 Am emailing you the binary to put on the downloads page

            • On April 30th, 2009, linuxcentre said:

              You’re a star….!

      • On April 30th, 2009, zinger5 said:

        Using the latest version of rtmpdump and get_ipalyer but having issues with downloads from five. They’re casuing rtmpdump to crash every time – BBC, ITV and CH4 all working ok though.

        • On April 30th, 2009, linuxcentre said:

          Have you tried reinstalling the whole Windows Automated Installer – it was overhauled a few days ago?

          • On April 30th, 2009, zinger5 said:

            yeah I initially tried it just using the –update and manually downloading rtmpdump 1.5 but when it wouldn’t work used the Automated Installer with the same results.

            • On April 30th, 2009, penguinny said:

              I am currently in touch with the author of rtmpdump trying to fix this crash. Can let you know once it is fixed. I have a version of rtmpdump that does not crash on my computer anymore and downloads from channel five, but I do not know enough sockets programming to be 100% sure that I fixed it correctly. Can send you my version if you would like to try it.

              • On April 30th, 2009, penguinny said:

                rtmpdump ver.1.5a is now available from the sourceforge, which should fix the channel 5 problem.

                • On April 30th, 2009, zinger5 said:

                  That’s great – just tried the new version and it seems to work now (you have to get the Win32 binary from rtmpdump’s Sourceforge repository though – the link in the Downloads section here is still the older one).

                  Good job – this is a great application. I hope some clever person finds a way to compile it for the Popcorn Hour media player – getting iPlayer working on it would be great.

                  zinger5

                  • On April 30th, 2009, linuxcentre said:

                    I’ve posted Windows and Linux/Fedora rtmpdump v1.5a on the download page now.

                  • On January 13th, 2010, Tim said:

                    Hi Zinger, I know this is a very long time ago, i have spent a considerable amount of time playing with the flvstreamer and get_iplayer for the QNAP nas and also now the PopCorn Hour. I have successfully got get_iplayer working on the MIPes processor, i still have a little way to go, but it’s pretty much there. I have 1 last issue with regards FFmpeg and then i will try and put some documentation together if you are interested.

                    Tim.

        • On May 1st, 2009, Philip said:

          The fix in 1.74 for Channel 4 doesn’t seem to be working very well – it looks like it picks a random (non-integer) number between 0 and 6, then looks in %user_agent for that number as a key, so it always gets undefined, and ends up sending no User-Agent header at all, and gets blocked by Ch4.

          (Apart from that, this is an excellent tool 🙂 )

          • On May 1st, 2009, Philip said:

            Oh, also the Referer is set to “$baseurl.’/programmes/catch-up” instead of “$baseurl/programmes/catch-up”. If I fix that and the UA, then I can successfully download the programme listings, though then it fails to download the actual programme.

            • On May 1st, 2009, linuxcentre said:

              Thanks, user agent selection bugfixed in v1.75.

              • On May 3rd, 2009, Philip said:

                Still a little bug: the code now says $user_agent_list[ int(rand($#user_agent_list))], so if (for example) @user_agent_list had two entries then $#user_agent_list == 1 and int(rand(1)) is always 0. So it’ll never pick the last entry in the list.

                (Easily fixed with int(rand($#user_agent_list+1)) or int(rand(@user_agent_list)) instead. And actually you can just say $user_agent_list[rand @user_agent_list] because array indexes automatically get rounded down. But, uh, I can’t work out where that feature is documented. Surely it must be somewhere…)

                • On May 4th, 2009, linuxcentre said:

                  Ah – yes – sorry – too much beer and tiredness in the last few days. I’ve fixed and tested that now in v1.77. The trick is to add 0.5.

                  • On May 4th, 2009, Philip said:

                    Hmm, that doesn’t seem like a great trick. If you have three items in the list and use int(rand(2+0.5)), there’s a 40% chance it’ll give 0, 40% it’ll give 1, and 20% 2, so it’s not very uniform. If you use int(rand(2)+0.5) instead, then it’s 25% 0, 50% 1, and 25% 2, so it’s still not very uniform. I think you really want to use int(rand(3)) instead 🙂 (and rand is defined such that it might give 2.999… but never precisely 3 so it’ll always work fine)

                • On May 5th, 2009, linuxcentre said:

                  Used a better way now. Now its nice and uniform 🙂 Thanks.
                  $uas = $user_agent_list[rand @user_agent_list];

          • On May 1st, 2009, fon said:

            The first time I have tried to download from the FIVE channel:

            INFO: Command: .\rtmpdump.exe -r “rtmpe://streamflv.download.five.tv/a800/o10/mp
            4:bthttp/five/downloadstore/secure/peppapig/movies/Peppa-Pig_S2_E1_Episode-1_FLV
            STREAM.mp4?e=1241195948&h=a962d8e0c885052f0aabcb9843d0e799” –resume –swfsize 8
            48259 –swfhash 75a4a2a2dc7e7c198478161bb1c8826721034f4f36bed7d23c6bf42910a500a1
            –protocol “3” -o “Downloads\/Peppa_Pig_Episode_1_C5115230001.partial.mp4.flv”
            1>&2
            RTMPDump v1.5a
            (c) 2009 Andrej Stepanchuk, license: GPL

            DEBUG: Parsing…
            DEBUG: Parsed protocol: 3
            DEBUG: Parsed host : streamflv.download.five.tv
            DEBUG: Parsed app : a800/o10
            DEBUG: Parsed playpath: mp4:bthttp/five/downloadstore/secure/peppapig/movies/Pep
            pa-Pig_S2_E1_Episode-1_FLVSTREAM.mp4?e=1241195948&h=a962d8e0c885052f0aabcb9843d0
            e799
            DEBUG: Setting buffer time to: 36000000ms
            Connecting …
            DEBUG: Protocol : RTMPE
            DEBUG: Hostname : streamflv.download.five.tv
            DEBUG: Port : 1935
            DEBUG: Playpath : mp4:bthttp/five/downloadstore/secure/peppapig/movies/Peppa-Pig
            _S2_E1_Episode-1_FLVSTREAM.mp4?e=1241195948&h=a962d8e0c885052f0aabcb9843d0e799
            DEBUG: tcUrl : rtmpe://streamflv.download.five.tv:1935/a800/o10
            DEBUG: app : a800/o10
            DEBUG: flashVer : LNX 9,0,124,0
            DEBUG: live : no
            DEBUG: timeout : 300 sec
            DEBUG: SWFSHA256:
            75 A4 A2 A2 DC 7E 7C 19 84 78 16 1B B1 C8 82 67 21 03 4F 4F 36 BE D7 D2 3C 6B F4
            29 10 A5 00 A1
            DEBUG: SWFSize : 848259
            ERROR: Connect, failed to connect socket. Error: 10065
            Failed to connect!
            WARNING: Failed to download file Downloads\/Peppa_Pig_Episode_1_C5115230001.part
            ial.mp4.flv via RTMP
            INFO: skipping flashhigh mode
            ERROR: Failed to download ‘Peppa Pig – Episode 1 (C5115230001)’

            Has anyone else had this error or is it just me? Does this socket error mean than the server is unavailable? Any suggestions gratefully received.

            • On May 1st, 2009, James said:

              Is there any hope of the 64bit Linux rtmpdump being posted up? The 32bit version doesn’t work for me and for some reason I can’t get the 64bit version to compile from source.

              • On May 2nd, 2009, keswick said:

                This is not good!
                Can anyone help?
                Iam trying to download peppa pig from five but i keep getting this message?
                I have the latest iplayer installer and have downloaded 1.5a of rtmp from scourceforge, am i doing something wrong?
                I copied and pasted latest rtmpdump in get_iplayer directory, is this the correctthing to do?
                cheers

                C:\Program Files\get_iplayer>get_iplayer –type=five –get 50444

                C:\Program Files\get_iplayer>perl.exe get_iplayer.pl –type=five –get 50444
                get_iplayer v1.75, Copyright (C) 2009 Phil Lewis
                This program comes with ABSOLUTELY NO WARRANTY; for details use –warranty.
                This is free software, and you are welcome to redistribute it under certain
                conditions; use –conditions for details.

                Matches:
                50444: Peppa Pig – Episode 13, five, tv,

                INFO: 1 Matching Programmes
                INFO: Checking existence of default version
                INFO: flashhigh modes will be tried for version default
                INFO: Trying flashhigh mode to download five: Peppa Pig – Episode 13
                WARNING: Required program .\rtmpdump.exe does not exist (see http://linuxcentre.
                net/getiplayer/installation and //linuxcentre.net/getiplayer/download)
                INFO: skipping flashhigh mode
                ERROR: Failed to download ‘Peppa Pig – Episode 13 (C5115230013)’

                C:\Program Files\get_iplayer>
                C:\Program Files\get_iplayer>get_iplayer –type=five –get 50444 –raw

                C:\Program Files\get_iplayer>perl.exe get_iplayer.pl –type=five –get 50444 –r
                aw
                get_iplayer v1.75, Copyright (C) 2009 Phil Lewis
                This program comes with ABSOLUTELY NO WARRANTY; for details use –warranty.
                This is free software, and you are welcome to redistribute it under certain
                conditions; use –conditions for details.

                Matches:
                50444: Peppa Pig – Episode 13, five, tv,

                INFO: 1 Matching Programmes
                INFO: Checking existence of default version
                INFO: flashhigh modes will be tried for version default
                INFO: Trying flashhigh mode to download five: Peppa Pig – Episode 13
                WARNING: Required program .\rtmpdump.exe does not exist (see http://linuxcentre.
                net/getiplayer/installation and //linuxcentre.net/getiplayer/download)
                INFO: skipping flashhigh mode
                ERROR: Failed to download ‘Peppa Pig – Episode 13 (C5115230013)’

                C:\Program Files\get_iplayer>

                • On May 2nd, 2009, Matt said:

                  Just upgraded to 1.75, can get itv, five and all bbc stuff, but ch4 just gives me:

                  get_iplayer v1.75, Copyright (C) 2009 Phil Lewis
                  This program comes with ABSOLUTELY NO WARRANTY; for details use –warranty.
                  This is free software, and you are welcome to redistribute it under certain
                  conditions; use –conditions for details.

                  INFO: Getting ch4 Programmes List
                  WARNING: Failed to get series list from from channel4 site
                  WARNING: Failed to get series list from from channel4 site
                  WARNING: Failed to get series list from from channel4 site

                  INFO: 0 Matching Programmes

                  Any ideas why???!!!

                  • On May 2nd, 2009, DavidC said:

                    Fix to 1.75 issue above.

                    — get_iplayer-1.75.orig Sat May 2 16:25:52 2009
                    +++ get_iplayer-1.75 Sat May 2 16:26:25 2009
                    @@ -1267,7 +1267,11 @@
                    my $ua = LWP::UserAgent->new;
                    $ua->timeout([$lwp_request_timeout]);
                    $ua->proxy( [‘http’] => $opt->{proxy} ) if $opt->{proxy};
                    – $ua->agent( $user_agent{$agent} );
                    + if (length($user_agent{$agent}) == 0) {
                    + $ua->agent( $agent );
                    + } else {
                    + $ua->agent( $user_agent{$agent} );
                    + }
                    $ua->conn_cache(LWP::ConnCache->new());
                    #$ua->conn_cache->total_capacity(50);
                    $ua->cookie_jar( HTTP::Cookies->new( file => $cookiejar, autosave => 1, ignore_discard => 1 ) ) if not $nocookiejar;

                    • On May 2nd, 2009, DavidC said:

                      Unmangle diff

                      — get_iplayer-1.75.orig Sat May 2 16:36:24 2009
                      +++ get_iplayer-1.75 Sat May 2 16:33:34 2009
                      @@ -1268,5 +1268,9 @@
                      $ua->timeout([$lwp_request_timeout]);
                      $ua->proxy( [‘http’] => $opt->{proxy} ) if $opt->{proxy};
                      – $ua->agent( $user_agent{$agent} );
                      + if (length($user_agent{$agent}) == 0) {
                      + $ua->agent( $agent );
                      + } else {
                      + $ua->agent( $user_agent{$agent} );
                      + }
                      $ua->conn_cache(LWP::ConnCache->new());
                      #$ua->conn_cache->total_capacity(50);

                      • On May 2nd, 2009, DavidC said:

                        Hm still mangled, but you get the idea

                        • On May 3rd, 2009, linuxcentre said:

                          thanks – I’ve hopefully fixed that prblem now.

                  • On May 2nd, 2009, Gary Calder said:

                    Five downloads are not working for me on Ubuntu 9.04, rtmpdump (v1.5a) just seems to hang afetr SWFSIZE:

                    ./get_iplayer –type=five –rtmpdump=’./rtmpdump’ –get 50187
                    get_iplayer v1.75, Copyright (C) 2009 Phil Lewis
                    This program comes with ABSOLUTELY NO WARRANTY; for details use –warranty.
                    This is free software, and you are welcome to redistribute it under certain
                    conditions; use –conditions for details.

                    Matches:
                    50187: Fifth Gear – Episode 1, five, tv,

                    INFO: 1 Matching Programmes
                    INFO: Checking existence of default version
                    INFO: flashhigh modes will be tried for version default
                    INFO: Trying flashhigh mode to download five: Fifth Gear – Episode 1
                    INFO: File name prefix = Fifth_Gear_Episode_1_C5143730001

                    INFO: Command: ./rtmpdump -r “rtmpe://streamflv.download.five.tv/a800/o10/mp4:bthttp/five/downloadstore/secure/fifthgear/movies/Fifth-Gear_S15_E1_Episode-1_FLVSTREAM.mp4?e=1241259330&h=709052a6c91019d9b7adb43eb1248a54” –resume –swfsize 848259 –swfhash 75a4a2a2dc7e7c198478161bb1c8826721034f4f36bed7d23c6bf42910a500a1 –protocol “3” -o “./Fifth_Gear_Episode_1_C5143730001.partial.mp4.flv” 1>&2
                    RTMPDump v1.5a
                    (c) 2009 Andrej Stepanchuk, license: GPL

                    DEBUG: Parsing…
                    DEBUG: Parsed protocol: 3
                    DEBUG: Parsed host : streamflv.download.five.tv
                    DEBUG: Parsed app : a800/o10
                    DEBUG: Parsed playpath: mp4:bthttp/five/downloadstore/secure/fifthgear/movies/Fifth-Gear_S15_E1_Episode-1_FLVSTREAM.mp4?e=1241259330&h=709052a6c91019d9b7adb43eb1248a54
                    DEBUG: Setting buffer time to: 36000000ms
                    Connecting …
                    DEBUG: Protocol : RTMPE
                    DEBUG: Hostname : streamflv.download.five.tv
                    DEBUG: Port : 1935
                    DEBUG: Playpath : mp4:bthttp/five/downloadstore/secure/fifthgear/movies/Fifth-Gear_S15_E1_Episode-1_FLVSTREAM.mp4?e=1241259330&h=709052a6c91019d9b7adb43eb1248a54
                    DEBUG: tcUrl : rtmpe://streamflv.download.five.tv:1935/a800/o10
                    DEBUG: app : a800/o10
                    DEBUG: flashVer : LNX 9,0,124,0
                    DEBUG: live : no
                    DEBUG: timeout : 300 sec
                    DEBUG: SWFSHA256:
                    75 A4 A2 A2 DC 7E 7C 19 84 78 16 1B B1 C8 82 67 21 03 4F 4F 36 BE D7 D2 3C 6B F4 29 10 A5 00 A1
                    DEBUG: SWFSize : 848259

                    • On May 3rd, 2009, Matt said:

                      If I search for a show on Five like The Gadget Show (get_iplayer –type=five gadget), the results I get are the episodes from the latest series (series 11). Series 10 is also available on the website in flash but doesn’t show up in the results and Series 9 is available in Windows Media Player. Why isn’t series 10 indexed?

                      • On May 3rd, 2009, linuxcentre said:

                        That is because I haven’t gotten around to doing older series for each programme. The older WMV versions aren’t supported by get_iplayer.

                        • On May 3rd, 2009, linuxcentre said:

                          I should have mentioned that you can use: –pid five: to get any of those other flash based episodes.

                        • On May 4th, 2009, Saf said:

                          Hi,

                          Great work on this. I’ve been using this since the 0.xx versions and it’s come so far.

                          Just the one question:

                          I use OSX (leopard) and is it possible to specify the location of rtmpdump and ffmpeg for getiplayer to remember? So that every time I put a ch4 or five download on, I don’t always have to use the commands –rtmpdump (location) –ffmpeg (location).

                          Using this method, it would remember where rtmpdump and ffmpeg is so we just have to put the download on and it will call for rtmpdump and ffmpeg in the locations which we have specified.

                          If that makes sense! Not sure I worded that very well!

                          Anyway, It’s a great program. Thanks for all the hard work you’ve put in to it.

                          Thanks,
                          Saf

                          • On May 4th, 2009, linuxcentre said:

                            You can use the –save option to save the current cmdline options (saved them in .get_iplayer/options)

                            • On May 4th, 2009, Saf said:

                              Hi,
                              Thanks. I didn’t use that because I assumed that it would try to use rtmpdump and ffmpeg for the normal iphone downloads as well and always automatically download the flash version.
                              But it works perfect. Should’ve tried it first instead of assuming!

                              Thanks,
                              Saf

                          • On May 4th, 2009, AudioBear said:

                            Is it me or if everytime I download something from CH4 I get ‘Countdown’… I’ve flushed the CH4 index
                            e.g.
                            ./get_iplayer –pid=ch4:2872163

                            Should be Henry VIII’s Lost Palaces Episode 1
                            http://www.channel4.com/programmes/henry-viiis-lost-palaces/catch-up#2872163

                            Or The 100 Greatest Kids TV Shows
                            http://www.channel4.com/programmes/the-100-greatest-kids-tv-shows/catch-up#2872100

                            ./get_iplayer –pid=ch4:2872100

                            More Countdown… any ideas
                            Countdown_s2009e70_2872100.mp4

                            • On May 4th, 2009, Alan said:

                              I believe that this is fixed in version 1.76 (earlier today).

                              get_iplayer –update

                              should do it

                              • On May 4th, 2009, Alan said:

                                Sorry, I meant version 1.77.

                            • On May 5th, 2009, keswick said:

                              The update 1.80v doesnt seem to work with ch4 or five searchs anymore!
                              i get this message ERROR: Invalid type ‘five’ specified. Valid types are: itv,tv,podcast,radio

                              can anyone help?

                              • On May 5th, 2009, linuxcentre said:

                                I’ve moved get_iplayer to be plugin based. You just need to run the update again (i.e. ‘get_iplayer –update’ or get_iplayer update from the menu in Windows version). Failing that try ‘get_iplayer –update –force’ which will update the script and plugins regardless of the current version number.

                              • On May 5th, 2009, AndrewM said:

                                This is a very minor thing, but you might want to reorder the numbering of CDNs for flashhigh and flashvhigh because akamai (corresponding to flashvhigh2 and flashhigh3) now definitely seems to be the default and indeed it has been the only flashvhigh or flashhigh CDN for all the programmes I’ve checked recently. (Of course I’m not suggesting that you remove the other CDNs, they might be used again for flashhigh or flashvhigh in the future. Interestingly though, flashaac1 (limelight) seems to be the preferred CDN for AAC versions of radio programmes.)

                                Also, I noticed that you’re currently using “http://www.bbc.co.uk/emp/9player.swf?revision=7978_8340” for the swfurl parameter for flashhd, flashvhigh and flashhigh and “http://www.bbc.co.uk/emp/9player.swf?revision=7276” for flashnormal, however the BBC website is currently using “http://www.bbc.co.uk/emp/9player.swf?revision=8812_8903”. I think if anyone looks at the log files it could look rather strange if requests are using a revision older than the website is using so you might want to update them.

                                swfurl for flashlow1 is currently set at “http://www.bbc.co.uk/emp/iplayer/7player.swf?revision=3897” (which returns a 404) and for flashlow2 at “http://www.bbc.co.uk/emp/9player.swf?revision=7978_8340”. I assume they both should use “http://www.bbc.co.uk/emp/9player.swf?revision=8812_8903”, but I don’t know how to access these streams on the BBC website so I’m not sure.

                                And for radio, flashaac1, flashaac2 and flashaudio also are using old revisions for swfurl.

                                • On May 5th, 2009, Andy said:

                                  Has anyone had any luck with the IT Crowd episode ‘Are We Not Men?’, with the PID 40439? All I get is ffmpeg spitting it back at me with:

                                  [flv @ 0x1002400]negative cts, previous timestamps might be wrong
                                  [NULL @ 0x1003400]error, non monotone timestamps 40 >= 40
                                  av_interleaved_write_frame(): Error while opening file
                                  WARNING: flv conversion failed – retaining flv file

                                  All of the others work fine… any clues? :o)

                                  • On May 6th, 2009, AudioBear said:

                                    Hi Andy, this programme does not seem to be available on the CH4 4OD website? I just tied it with the pid and am getting the default ‘Countdown’ – so no pid or bad pid?

                                    Also that pid looks a little short, for a latter episodes of The IT Crowd programmes they are:
                                    Calendar Geeks: 2873354
                                    Friendface: 2872680
                                    Tramps Like Us: 2872120

                                    I can download at least one of these (2872120)
                                    /get_iplayer –pid=ch4:2872120

                                    Also are you using the latest version of get_iplayer (1.82 and also the latest rtmpdump 1.5a)?

                                    • On May 6th, 2009, Andy said:

                                      Hi AudioBear,

                                      Yes, I’m on get_iplayer v1.82 and rtmpdump v1.5a. Interestingly, the episode has vanished from the Channel 4 On Demand system now, so perhaps it was left on by mistake?

                                      Thanks for looking into it, though!

                                      Cheers,

                                      Andy.

                                    • On May 6th, 2009, AudioBear said:

                                      Actually Andy, I just looked at the video I downloaded and it is ‘Countdown’ again. Looks like a problem either with get_iplayer or CH4.

                                      Phil, all the meta data is correct but the video is ‘Countdown’…
                                      /get_iplayer –pid=ch4:2872120

                                      • On July 28th, 2009, Leo Gee said:

                                        I get the following error message when trying to search or download from ch4 or five. Using –type=ch4 or –type=five :
                                        “Invalid file type five specified. Valid file types are: livetv, itv, tv, liveradio, podcast, radio.

                                        I have the latest version of iplayer 2.11 and have updated plugins.

                                        What am I doing wrong?

                                        • On July 28th, 2009, linuxcentre said:

                                          ch4, hulu and five are no longer supported.

                                          • On July 28th, 2009, Leo Gee said:

                                            OK, I read the update link at the top and understand. Hopefully these options will come back sometime.

                                            • On July 29th, 2009, AudioBear said:

                                              Lee, for five at least rtmpdump is your friend 😉

                                        • 4 old comments are not displayed. Click to display all comments

                                        WordPress Themes