Downloading Multiple Files with .Net 3.5, 4 and 4.5

0 Comments

Lately I’ve been developing quite a lot of simple download managers. What I needed was a verysimple file downloader with these requirements:

  • Concurrency: 1 thread is just not enough. I wanted to download multiple files at once. On the other hand I wanted to be able to change the number of concurrent downloads.
  • Simplicity: I didn’t want to nor did I have the time to go into details to create a complex download manager. I wanted to use as much framework classes as possible.
  • Logging Support: I wanted to be able to support the logging feature. I wanted it to be both simple and flexible enough and be as abstract as possible.

The code that I’ll be demonstrating will be in C# but you could easily convert it for other .Net languages.

First, for logging, I created an interface for logger, named (surprise) ILogger. Here’s the code:

[code lang="csharp"]
    public interface ILogger
    {
        void WriteLog(string message);
        void WriteDebug(string message);
        void ReportProgress(int percent);
    }
[/code]

The interface requires 3 functions for writing logs, debug messages and reporting progress in percent (for ProgressBars, etc.).

And implementation of this interface (which I have used in my apps) is an event-based logger. This class will fire up an event every time a log message is written and using it is quite simple. I think you’ll get it, so no more explanations Winking smile

[code lang="csharp"]
    class EventBasedLogger : ILogger
    {
        public delegate void LogHandler(string message);
        public event LogHandler Log;
        public event LogHandler Debug;

        public delegate void ProgressHandler(int percent);
        public event ProgressHandler Progress;

        public void WriteLog(string message)
        {
            if (Log != null)
            {
                Log(message);
            }
        }

        public void WriteDebug(string message)
        {
            if (Debug != null)
            {
                Debug(message);
            }
        }

        public void ReportProgress(int percent)
        {
            if (Progress != null)
            {
                Progress(percent);
            }
        }
    }
[/code]

OK. One more thing. Every download item is represented with a very simple class:

[code language="csharp"]
    public class DownloadItem
    {
        public Uri Address { get; set; }
        public string Path { get; set; }
    }
[/code]

It has 2 properties to show the Uri (mostly internet URL) and the path in which the file will be saved.

Now, to the fun part. The downloader class’ code for the new versions of .Net is like this:

[code language="csharp"]
    public class FileDownloader
    {
        private int processed = 0;
        private int initialCount = 0;

        public ILogger Logger { get; set; }
        private void OnProgress(int percent)
        {
            if (Logger != null)
            {
                Logger.ReportProgress(percent);
            }
        }

        public void StartDownload(IEnumerable&lt;DownloadItem&gt;<downloaditem> items, CancellationToken ct)
        {
            initialCount = items.Count();

            Parallel.ForEach
                (
                items,
                new ParallelOptions { MaxDegreeOfParallelism = 8 },
                file =&gt;
                {
                    if (ct.IsCancellationRequested)
                        ct.ThrowIfCancellationRequested();

                    try
                    {
                        if (!File.Exists(file.Path))
                        {
                            try
                            {
                                using (var cli = new WebClient())
                                {
                                    cli.DownloadFile(file.Address, file.Path);
                                }
                            }
                            catch (Exception)
                            {
                            }
                        }

                        int pro = Interlocked.Increment(ref processed);
                        double percent = 100.0 * pro / initialCount;
                        OnProgress((int)percent);
                    }
                    catch (Exception)
                    {
                        if (File.Exists(file.Path))
                            File.Delete(file.Path);
                    }
                }
                );
        }
    }
[/code]

But older versions of .Net don’t have the Parallel class. So what then? Well, I created another class just for this purpose:

[code language="csharp"]
    public class FileDownloader
    {
        public ILogger Logger { get; set; }
        private void OnProgress(int percent)
        {
            if (Logger != null)
            {
                Logger.ReportProgress(percent);
            }
        }

        private void OnLog(string message)
        {
            if (Logger != null)
            {
                Logger.WriteLog(message);
            }
        }

        private void OnDebug(string message)
        {
            if (Logger != null)
            {
                Logger.WriteDebug(message);
            }

        }

        private Queue<downloaditem> queue = new Queue<downloaditem>();

        private const int maxThreads = 8;

        private int processed = 0;

        private int processesing = 0;

        private int totalCount = 0;

        private string lockObj = "";

        //this is NOT the best way to cancel async operations
        // this is just for simplicity
        public bool CANCEL = false;

        public void StartDownload(Queue<downloaditem> queue)
        {
            OnLog("Starting download... File count: " + queue.Count);
            this.queue = queue;
            this.totalCount = queue.Count;
            StartDownload();
        }

        private void StartDownload()
        {
            if (CANCEL || queue.Count == 0 || Interlocked.Equals(processed, totalCount))
            {
                if (CANCEL)
                    OnLog("Download canceled.");
                else
                    OnLog("Finished download.");
                return;
            }

            lock (lockObj)
            {
                for (int i = 0; !Interlocked.Equals(processesing, maxThreads) &amp;&amp; (queue.Count &gt; 0) &amp;&amp; !CANCEL; i++)
                {
                    var file = queue.Dequeue();

                    try
                    {
                        if (!File.Exists(file.Path))
                        {
                            //start downloading the file
                            var cli = new WebClient();
                            OnDebug(string.Format("Downloading {0} to {1}", file.Address, file.Path));
                            cli.DownloadFileCompleted += cli_DownloadFileCompleted;
                            Interlocked.Increment(ref processesing);
                            cli.DownloadFileAsync(file.Address, file.Path);
                        }
                        else
                            OnDebug("Skipping file " + file.Path);
                    }
                    catch (Exception)
                    {
                        OnLog("Error downloading file from " + file.Address);
                        //Do whatever you want. Maybe write a debug message, stop download, etc.
                    }
                }
            }
        }
[/code]

GitHub

The code for this tutorial (with comments, demo, etc.) is available at my GitHub:

https://github.com/alirezanoori1986/FileDownloader

Go have fun and let me know if you liked this approach in the comments.