Question:
Let's say I'm downloading pictures in 100 threads. If any iteration caused an exception – how to repeat it again?
Parallel.For(0, newLst.Count, new ParallelOptions { MaxDegreeOfParallelism = 100 }, (i) =>
{
DownloadImage(newLst[i]);
});
Answer:
Do DownloadImageAsync
, which is not in Parallel.For
, but simply by limiting the number of simultaneously executing tasks, for example, using this code:
public static IEnumerable<Task<TTask>> ForEachAsync<TItem, TTask>(
this IEnumerable<TItem> source, Func<TItem, Task<TTask>> selector,
int degreeOfParallelism)
{
Contract.Requires(source != null);
Contract.Requires(selector != null);
// We need to know all the items in the source before starting tasks
var tasks = source.ToList();
int completedTask = -1;
// Creating an array of TaskCompletionSource that would holds
// the results for each operations
var taskCompletions = new TaskCompletionSource<TTask>[tasks.Count];
for(int n = 0; n < taskCompletions.Length; n++)
taskCompletions[n] = new TaskCompletionSource<TTask>();
// Partitioner would do all grunt work for us and split
// the source into appropriate number of chunks for parallel processing
foreach (var partition in
Partitioner.Create(tasks).GetPartitions(degreeOfParallelism))
{
var p = partition;
// Loosing sync context and starting asynchronous
// computation for each partition
Task.Run(async () =>
{
while (p.MoveNext())
{
var task = selector(p.Current);
// Don't want to use empty catch .
// This trick just swallows an exception
await task.ContinueWith(_ => { });
int finishedTaskIndex = Interlocked.Increment(ref completedTask);
taskCompletions[finishedTaskIndex].FromTask(task);
}
});
}
return taskCompletions.Select(tcs => tcs.Task);
}
Now you can do this:
var tasks = newLst.ForEachAsync(n => DownloadImageAsyncWithRetry(n, retryCount), 100).ToList()
And just shove the logic of retraining into DownloadImageAsyncWithRetry
:
public Task<Result> DownloadImageAsyncWithRetry(input)
{
var tsk = DownloadImageAsync(input);
// Тут все зависит от того, каким именно образом определяется неудача.
// Если это исключение, то вешаем ContinueWith, если это код возврата,
// то проверяем его и пробуем повторить запрос.
}
Yes, Parallel.For
is not a good idea in this case, since it is intended primarily for CPU Intensive operations, but here it is clearly IO Intensive. In this case, it is logical to make the method itself asynchronous, which will pull the asynchronous API for loading pictures.
Z.Y. Take the ForEachAsync code from here .