|Chat||Windows build status||Linux build status|
The Microsoft Cognitive Toolkit (https://cntk.ai) is a unified deep learning toolkit that describes neural networks as a series of computational steps via a directed graph. In this directed graph, leaf nodes represent input values or network parameters, while other nodes represent matrix operations upon their inputs. CNTK allows users to easily realize and combine popular model types such as feed-forward DNNs, convolutional nets (CNNs), and recurrent networks (RNNs/LSTMs). It implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers. CNTK has been available under an open-source license since April 2015. It is our hope that the community will take advantage of CNTK to share ideas more quickly through the exchange of open source working code.
If you prefer to use latest CNTK bits from master, use one of the CNTK nightly packages:
You can learn more about using and contributing to CNTK with the following resources:
CNTK is in active use at Microsoft and constantly evolving. There will be bugs.
You can find more news on the official project feed
2018-04-16. CNTK 2.5.1
Repack CNTK 2.5 with third party libraries included in the bundles (Python wheel packages)
2018-03-15. CNTK 2.5
Change profiler details output format to be
Enable per-node timing. Working example here
import cntk as C C.debugging.debug.set_node_timing(True) C.debugging.start_profiler() # optional C.debugging.enable_profiler() # optional #<trainer|evaluator|function> executions <trainer|evaluator|function>.print_node_timing() C.debugging.stop_profiler()
Example profiler details view in
CPU inference performance improvements using MKL
1BitSGD incorporated into CNTK
1BitSGDsource code is now available with CNTK license (MIT license) under
1bitsgdbuild target was merged into existing gpu target
New loss function: hierarchical softmax
Distributed Training with Mulitple Learners
CNTKBinaryFormatdeserializer when crossing sweep boundary
misc.converter.py, which prevents converting complex networks.
OptimizedRNNStackoperator (LSTM only).
LayerNormalizationlayer using ONNX
LeakyReLu(argument ‘alpha’ reverted to type double).
2018-02-28. CNTK supports nightly build
If you prefer to use latest CNTK bits from master, use one of the CNTK nightly package.
Alternatively, you can also click corresponding build badge to land to nightly build page.
2018-01-31. CNTK 2.4
top_koperation: in the forward pass it computes the top (largest) k values and corresponding indices along the specified axis. In the backward pass the gradient is scattered to the top k elements (an element not in the top k gets a zero gradient).
gatheroperation now supports an axis argument
expand_dimsoperations for easily removing and adding singleton axes
ones_likeoperations. In many situations you can just rely on CNTK correctly broadcasting a simple 0 or 1 but sometimes you need the actual tensor.
depth_to_space: Rearranges elements in the input tensor from the depth dimension into spatial blocks. Typical use of this operation is for implementing sub-pixel convolution for some image super-resolution models.
space_to_depth: Rearranges elements in the input tensor from the spatial dimensions to the depth dimension. It is largely the inverse of DepthToSpace.
sumoperation: Create a new Function instance that computes element-wise sum of input tensors.
softsignoperation: Create a new Function instance that computes the element-wise softsign of a input tensor.
asinhoperation: Create a new Function instance that computes the element-wise asinh of a input tensor.
log_softmaxoperation: Create a new Function instance that computes the logsoftmax normalized values of a input tensor.
hard_sigmoidoperation: Create a new Function instance that computes the hard_sigmoid normalized values of a input tensor.
element_xorelement-wise logic operations
reduce_l1operation: Computes the L1 norm of the input tensor’s element along the provided axes.
reduce_l2operation: Computes the L2 norm of the input tensor’s element along the provided axes.
reduce_sum_squareoperation: Computes the sum square of the input tensor’s element along the provided axes.
image_scaleroperation: Alteration of image by scaling its individual values.
Reshapeop to handle
producer_versionfields to ONNX models.
padsatrribute is specified in ONNX
InputVariablewith only one batch axis.
Transposeop to match updated spec.
Poolingops to match updated spec.
Convolutionop will change for groups > 1. More optimized implementation of group convolution is expected in the next release.
Halide Binary Convolution
Cntk.BinaryConvolution.so/dlllibrary that can be used with the
netoptmodule. The library contains optimized binary convolution operators that perform better than the python based binarized convolution operators. To enable Halide in the build, please download Halide release and set
HALIDE_PATHenvironment varibale before starting a build. In Linux, you can use
./configure --with-halide[=directory]to enable it. For more information on how to use this feature, please refer to How_to_use_network_optimization.
2018-01-22. CNTK support for CUDA 9
CNTK now supports CUDA 9/cuDNN 7. This requires an update to build environment to Ubuntu 16/GCC 5 for Linux, and Visual Studio 2017/VCTools 14.11 for Windows. With CUDA 9, CNTK also added a preview for 16-bit floating point (a.k.a FP16) computation.
Please check out the example of FP16 in ResNet50 here
Notes on FP16 preview:
To setup build and runtime environment on Windows:
vs_community.exe --add Microsoft.VisualStudio.Workload.NativeDesktop --add Microsoft.VisualStudio.Workload.ManagedDesktop --add Microsoft.VisualStudio.Workload.Universal --add Microsoft.Component.PythonTools --add Microsoft.VisualStudio.Component.VC.Tools.14.11
cmd /k "%VS2017INSTALLDIR%\VC\Auxiliary\Build\vcvarsall.bat" x64 --vcvars_ver=14.11
To setup build and runtime environment on Linux using docker, please build Unbuntu 16.04 docker image using Dockerfiles here. For other Linux systems, please refer to the Dockerfiles to setup dependent libraries for CNTK.
2017-12-05. CNTK 2.3.1 Release of Cognitive Toolkit v.2.3.1.
CNTK support for ONNX format is now out of preview mode.
If you want to try ONNX, you can build from master or
pip install one of the below wheels that matches your Python environment.
For Windows CPU-Only:
For Windows GPU:
You can also try one of the below NuGet package.