Petr Houška's devblog

SWE notes 02: SIFT - detect features regardless of scale without DNNs

Sat, 30 Apr 2022 21:53:40 +0000

SIFT: Scale invariant feature transform

Interest point:
- Rich content (brightness and color variation, …)
- Well defined representation for matching comparison with other points
- Well defined position in the image
- Scale, orientation, brightness, … invariant
What are good interest points?
- Not edges -> not descriptive / unique enough
- Corners only good for simpler images
- “Blobls” actually relatively good: location, orientation, size & possible to assign signature
Detecting blobs
- Detecting edges: first/second derivative of gaussian convolution (removes noise)
  - Extrema locations correspond to position of a blobs (edges on either side)
  - The larger the extrema the more prominent blob
- Changing sigma (for the gaussian): changing detection scale (Detecting Blobs SIFT Detector 6:20)
- Try multiple sigmas -> create stack of feature maps, each corresponding of trying to find blobs at different scale
Extracting interest points
- Get stack of feature maps per blob scale
- Compute differences of all two adjacent scale feature maps (smaller and bigger)
- Find extrema across all difference-featuremap featuremaps (3d max operator; 2d across space, 1d across scales)
- Filter one only high extrema (threshold)
- -> SIFT interest points
Scale invariance:
- We know the scale of interest points -> rescale them
Orientation invariance:
- For every pixel compute gradient (edge)
- Look just at orientation (magnitude is about lightning), create histogram
- Take principal (largest) orientation and use it to normalize location (rotate the patch through the orientation)
SIFT descriptor
- Create histogram per normalized (orientation, scaling) point of interest (usually divided into 4 subplots)
- Distance between histograms can be normalized correlation / L2 / …
Allows many applications: s.a. matching features from one picture to another picture (different scale/orientation, …)

Deep learning notes 11: All is Kernels - but not those in ring0

Tue, 22 Mar 2022 21:26:37 +0000

Deep Networks Are Kernel Machines

Deep learning success is attributed to automatic discovery of new data representation
Paper’s proposal:
- Deep learning training: stores training data
- Deep learning inference: compares with stored data -> outputs closest -> ~kernel machine
- -> Deep network weights ~ superposition of training examples
- -> Network architecture incorporates knowledge of target function into kernel (similarity function)
Kernel machine: y = g ( sum_i: ai * K(x, xi) + b)
- K: similarity kernel function
- xi: remembered training set
- ai: gold labels
- -> essentially global nearest neighbor classifier
- The biggest question is how to choose the Kernel function
Gradient descent NN is equivalent to path-kernel Kernel machine
- Two datapoints are similar if gradients of the (network) output w.r.t to weights are similar (similar - inner product)
- Gradient of the (network) output w.r.t to weights: for particular input xi, how to change the weights to e.g. increase output
- Classify a datapoint
  - Retrace along all versions of the network (after each GD update)
  - Classify by whichever training datapoints had similar effect on the network (sensitivity) over the course of training
  - Get output of initial model and accumulate differences made by each GD step -> output of final
- Proof datapoint x, its classification y, training datapoints xi -> yi, weights wj, loss L:
  - Change in output for x w.r.t to steps of GD:
  - dy/dt = sum_j..d dy/dwj * dwj/dt | GSD’ed NN outputs change over time
  - = sum_j..d dy/dwj * (-dL/dwj) | Each wight changes according to Loss derivative
  - = sum_j..d dy/dwj (-sum_i..m dL/dyi * dyi/dwj) | Loss derivative over all data is sum over each data
  - = -sum_i..m dL/dyi * sum_j..d dy/dwj * dyi*dwj| Chain rule
  - = -sum_i..m L'(yi, yi) K_{f, w(t)}(x, xi) | re-arrange
  - => y = y0 - Integral_t sum_i..m L'(yi, yi) K_f,w(t) (x, xi) dt | output of NN y=
  - …some normalization…
  - = sum_i..m ai * K(x, xi) + b | output of Kernel machine
- Notes: ai, b depends on x,
Another way of looking at DNNs, also connects to boosting
- General statement: all learning methods will incorporate data

Kernels!

Refresher on concepts:
- Kernel functions: symmetric, positive semi-definitive ~ similarity measure
- Kernel matrix (e.g. gram matrix): pairwise distances of all points in dataset
Kernel: get kernel matrix without doing explicit feature map expansion on each datapoint and then dot product
- E.g. using simple elementwise kernel function -> way cheaper (or even actually possible) to evaluate
E.g.: Polynomial expansion with polynomial dimensionality p and datapoint dimensionality d:
- Full expansion: each datapoint d expanded to d*p (could be worse, inf. for other kernels), then dot product on these
- Kernel method: immediately computes the same kernel matrix out of two raw d-dim inputs
Hilbert spaces: imagine vector spaces but with generalized base vectors (functions, polynomials, …)
- Vector space needs to be linear and have scalar inner product
- ~Space of functions where the point evaluation functional is linear and continuous
- When converging in the function space -> also converging in outputs
Reproducing kernel, F set of functions fi: E->C, F is Hilbert space; K: ExE->C is reproducing kernel when:
- K(., t) € H, Vt€E
- <fi, K(., t)> = fi(t), Vt€E, Vfi € H | the value of fi(t) is reproduced by inner product of fi with K(., t)
- Reproducing Kernel Functions | IntechOpen, 7.pdf (berkeley.edu), rkhs2.pdf (berkeley.edu), optimization - Connection between SVM and Representer theorem - Cross Validated (stackexchange.com), functional analysis - Connection between Representer Theorem and Mercer’s Theorem? - Mathematics Stack Exchange
- Two reproducing kernels for two points -> equivalence between inner product of their reproducing kernels and its single evaluation
  - E.g.: <k(.,x), k(.,x')> = k(x, x') | for <f, g> = sum_i,j ai * bj * k(xi, xj) | H = ( f(x) = sum_i..m ai * k(xi, x) )
- Every reproducing kernel induces unique reproducing-kernel Hilbert space, and vice versa
- Every reproducing kernel is positive definite, and every positive definite kernel function defines unique reproducing kernel Hilbert space
Kernel methods ~ soft nearest neighbor
Any matrix of inner products (style transfer, attention, …) can be exploited through kernel lens
Various ways of looking at things
- SVM: Learn global separation planes
- DNN: Learn sequence of processing
Ridge regression with kernels: RBF, …
- Does regression in transformed space instead of original efficiently
- Seems to work good because regressing in those spaces seems to be good/better than original spaces
Representor theorem:
- ~Optimal function f*: x->y can be represented as linear comb. on the kernel functions k(x1, .), k(x2, .) of the dataset
- f*(.) = sum_1..n ai * k(., xi)
  - To get optimal ai coefficient: Take kernel products of our data (kernel matrix) & linear regression
  - Across all training points: find combination of similarities with all others that produces lowest error to its output
- Even assuming all possible datapoints, solution still lives in linear combination of only points from dataset
- Kernel method allows us to do this effectively -> coefficient on datapoints (their kernels) instead of their feature(s| maps)
  - Datapoints are the model
Usages:
- Style transfer: gram matrix in style
- Super-resolution: similarity patches
You could view DNN as implicit kernel method, indirectly also learns the similarity matrix

Deep learning notes 10: Diffusion models - noise to nice in few steps

Sun, 20 Feb 2022 22:10:35 +0000

DDPM - Diffusion Models Beat GANs on Image Synthesis

Input + sampled little bit of noise; repeated multiple times (~1000s) -> pure noise
- x_0 = x; noise(x_t+1|x_t); process input image from data distr.: x, applied noise(…) multiple times -> image of noise
If we could invert this process -> generative model: random normal noise image -> original image
- Learn to undo one “little bit of noise” step at a time: distribution noise(x_t-1|x_t)
- Sample random noise image, undo noise 1000s times (each time get one step cleaner image) -> sample clean data distr.
- Reversal gives us process of normal noise to data distribution
Noising up: q: noise(x_t+1|x_t) well-defined process of adding noise from Normal distribution
- Each step depends only on output of previous step
- Added noise has diagonal covariance matrix, is centered at last sample but down-scaled
- Given large T and well behaved schedule, last step is nearly isotropic Gaussian distribution
- Produces vast amount of data pairs of x_t-1, x_t
Denoising: p: noise(x_t-1|x_t) requires entire data distribution -> approximated via neural network
- Reversal doesn’t predict single image but a whole distribution of images (that could’ve been previous step)
- The output distribution is assumed to be gaussian (mean, covariance)
- The gaussian distribution assumption is maintained for small noise-up steps
Combination of p and q is ~VAE (variational auto-encoder) -> just train it
- The true distribution can be easily computed out of the both known training pairs
- Loss forces the denoising network predicted distribution to be close do the true distribution
The predicted covariance can be either statically set (actually doesn’t work super-bad) or also predicted
- If fixed: can be fixed based on the forward noise-up step parameters
Combination of two loss functions are used, for stability also resampled (early noise-up steps are more impactful)
- Simple objective: L2 difference between true and predicted picture / noise
- Variational loss: proper KL divergence VAE loss, including variance, …
Class-label guided generation can improve performance
- Train class classifier not only for clean images, but also for noised images -> use them to steer the generation
- Analogous to ~GANs; ~shifts the predicted distribution of the step-denoised-images to where specified label is likelier
Idea: Have GANs that have multiple discriminators along the path from noise to final image ~ merge these two approaches

Autoregressive Diffusion Models

New type of auto-regressive models: variables can be decoded in arbitrary order
Autoregressive models: produces tokens (words/patches/pixels) one after another
- E.g. First word, based on priming with it a second word, based on first two a third, …
- Usually a fixed order, for sentences starts with a first one, …
- Repeat until the whole output has been generated
ARDMs: Don’t have to go first to last, order could be arbitrary
- Can also produce multiple tokens at once reducing number of steps for lower accuracy
At the beginning all tokens are initialized (?randomly/zero?)
- DNN (usually transformer) processes them -> per token output (e.g. distribution over categories)
- A portion of them are sampled and decoded (argmax for categorization) -> concrete outputs for few tokens
- Concrete outputs replace random inputs for the sampled tokens, DNN, new subset of tokens are decoded, …
- Repeat until all tokens are sampled & set
~Similar to BERT
- Trained with random word within sentence masking -> predicts distribution over words for masked tokens
- Training is similar to BERT, just with non-fixed blanked tokens ratio
During training: mask a portion of tokens, average losses for all at once
- Sampling one timestep of one ordering where we decode & loss all of the remaining/maked tokens
- Left to right allows taking only the next (one) tokens’s loss -> less noisy
Why can’t we sample all tokens at once?
- Tokens aren’t independent -> argmax on one token (collapsing its distribution) influences other tokens’ disr.
- Sampling multiple at once is faster (less steps necessary) but possibly less ideal outputs
Extensions:
- Tokens could be re-sampled
- Multiple pixels at a time can be sampled -> to get the order/token groups dynamic programming
Initially sample only more rough values (e.g. out of few colors), only later revisit & predict specific color

Deep learning notes 09: Grokking - overfit into generalization

Sat, 20 Nov 2021 15:49:59 +0000

Grokking: Generalization beyond Overfitting on small algorithmic datasets

Neural network suddenly generalizes way beyond the point of overfitting with proper regularization
- Train accuracy rises with training steps, eventually NN overfits on training data, training continues…
- After orders of magnitude more steps, the network suddenly generalizes and test accuracy shoots up as well
Similar to the idea of double descent, just with training steps instead of number of parameters
In DD - when trained to convergence the relationship between test accuracy and number of parameters
- With higher capacity models test accuracy first increases as the model is becoming capable
- Then it starts going down when the network is big enough to remember dataset -> overfitting
- Increasing the number of parameters further leads to increase of accuracy again, surpassing previous best
- Interpretation: at some point there are enough parameters to nicely match all datapoints but smoothy (with proper regularization)
This paper dataset: variables + binary operation (e.g. polynomial operations) without any noise
- Dataset is a table of all pairs of variables + the result of the operation
- The network predicts the result of the operation for specific variables (portion blanked for trained data)
- Dataset is very specific & without noise, on real world issues the phenomena is hard to induce / see
Multiple variables of the dataset
- Size of the dataset
- Complexity of the operation
- Train dataset ratio
Training accuracy shoots to 100 % soon (10^2 steps), test accuracy to 100 % also but later (10^5)
- The higher the train set ratio, the faster the snap on test accuracy happens
- The easier the operation is (e.g. there are symmetries) the easier it happens
- The bigger the dataset, the harder it is to induce the phenomena
Weight decay seems to be very important to make the phenomena appear faster
- ? Prefers simple solutions vs remembering whole dataset
- So many train steps that a good solution is eventually discovered & then preferred because ^^
- Maybe weight decay is good-ish but not the best regularization
Visualization of the weights (t-SNE) shows structure that could be interpreted via the operation

Deep learning notes 08: Fastformer - additive or static (self)attention?

Mon, 11 Oct 2021 20:12:40 +0000

Fastformer - Additive attention can(not) be all you need

Modeling pairwise interaction between all pairs of tokens is expensive
Fastformer promises to use “additive attention” that’s linear in complexity via tokens-global aggregation
Presented in terms of queries, keys, values but could be just in terms of n (in this case 3) columns: a, b, …, z
Computation goes sequentially, starts with computing the output of the second column, then third, …, last
- For each column, create per-token input values, e.g. a1..an, b1…bn; the same way q,k,v are produced in transformer
- For computing the per-token outputs of second column Bi, start with Ai = ai
- For each Ai value, produce αi weight via softmax after transformation with learned wa, αi= exp(wa*Ai)/∑exp(wa*Aj)
- Produce global A as weighted average of Ai, A = ∑ αi * Ai
- The output of column b is then pointwise multiplication, Bi = bi x A
- In case there’s column c, we aggregate Bi to a single B, pointwise multiply with ci to get Ci
Still essentially quadratic i=0..n: Bi = bi x A = bi x ∑ αi * Ai = ∑ bi x αi * Ai
- Given there’s no softmax -> global a can be computed first -> linear in computation
The aggregation weights αi are essentially self-attention with per-column/layer static learned query wa
- Also could be viewed as soft classification according to learned static separation boundary vector wa
No information sharing between tokens apart from pointwise multiplication between global aggregate of prev. column
- Not really a proper attention; sort-of static query self-attention in the aggregation step
- It is statically learned what sort of tokens each layer/column should globally attend to; not dynamic per each token
- Good for tasks with global information, e.g. topic classification
Seems to just be framed in terms of the words of attention mechanism
In practice fast and with relatively good results on certain NLP tasks

Deep learning notes 07: PonderNet - Learn when to stop thinking

Fri, 27 Aug 2021 19:49:51 +0000

PonderNet - Learn when to stop thinking

Recurrent(ly run) network that can decide when to stop computation
In standard NN the amount of computation grows with size of input/output and/or is static; not with problem complexity
End-to-end trained to compromise between: computation cost (# of steps), training prediction accuracy, and generalization
- Halting node: predicts probability of halting on conditional of not halting before
- The rest of the network can be any architecture (rnn, cnn, transformer, …, capsule network, …)
Input x; x processed to hidden state h_i; processed via s(...) function; (h_i+1, y_i, λ_i) = s(h_i)
- Each steps returns next hidden state (h_i+1), output (y_i), and probability of stopping (λ_i)
- Probability to stop at step n: p_n = λ_n * TT_1..n-1 (1- λ_i)
At inference λ is used probabilistically (i.e. the probability is sampled)
Training loop:
- Input x into encoder, get h_0, … unroll the network for n steps regardless of λ
- Consider all outputs at the same time; loss = p_1*L(y_1)+p_2*L(y_2)+...+p_n*L(y_n)
- -> Possibly unstable -> two goals: make y_i better or make p_i smaller
- Regularization for KL(p_i || geometricDistirbution(λp)) -> forces lambdas to be similar to hyperparameter
Contrast vs ACT:
- Considers the output a weighted average of outputs: loss = L(p_1 * y_1 + ... + p_n * y_n)
- Early results need to be compatible with later results
- Less dynamic; needs more steps in experiments, worse in extrapolation
- Pondernet correctly needs more steps for more complex problems

Deep learning notes 06: Part-whole hierarchies with GLOM

Tue, 13 Jul 2021 20:34:27 +0000

GLOM: How to represent part-whole hierarchies in a neural network

Idea paper, not an actual implementation
- How does fixed architecture parse various pictures into part/whole hierarchy that’s different for each input
  - E.g. car is made of cabin, motor, …, cabin out of windows, doors, …
Dynamic image parsing sort-of handled by capsule networks
- First layer capsules represent/recognize lowest level features; capsule for window, door, …; second layer cabin, …
- Door and window activates cabin capsule, …
- ~discretization (active/nonactive) over implicit feature hierarchy of CNNs
GLOM architecture: large number of same weights columns, one for each spatial location
Each column is stack of spatially local autoencoders
- Each vertically divided into multiple (~5) levels
- Each level represents patch of image at different resolution/abstraction level
  - Cat’s ear -> furr, part of ear, cat’s head, cat, … ; neck -> furr, part of neck, cat’s neck, cat
  - All locations of cat’s ear will have similar second level activation, all locations in image similar last level activation
- At each level the activation is embedding vector of that feature at that location ~ CNNs but differently implemented
Communication/inference is iterative
- Between levels (layers) of each column through explicit neural networks
- Between columns through attention mechanism
- Iterative approach/eventual consistency forces all locations of a feature to share
For layer l, location x, timestep t+1 embedding is:
- e_t+1,l,x = e_t,l,x + f_td(e_t,l+1,x) f_bu(e_t,l-1,x) + <acrossLoc::below>
  - Last timestep + through NN (f_td, f_bu) above and below level
  - NN functions (f_td, f_bu) weights shared for the same level of all locations
- Positional encoding added to each input (similar to transformers)
Through message passing all locations for certain feature (e.g. Cat’s ear locations) converge on ~appropriate level activation
- Multiple locations sharing n-th level activation -> an island
- Following islands from topmost level to the bottom gives us parse hierarchy
  - The higher the bigger features -> the bigger islands; topmost: one island represents the whole image: class
For across-location/cross columns information sharing: attention over the same level of all columns
- Attends not using keys/queries but similarity: Instead of sm(QK^T)V -> sm(XX^T)X
- -> attends within islands -> converges towards clustering -> similar vectors forced toward similar vectors
Issue: on lower level similar things can share information even if they are in different parts of parse tree higher
- Possible solution: Module attention based on closeness in higher levels of parse tree (higher levels of columns) as well
- ~the further the less influential: sm(SUM_k=0..L-l λ^k X_l+k * X_l+k^T)X
Iterative algorithm, eventual consensus -> embeddings also update -> doesn’t discover clusters but creates them
Designed decisions:
- Locations per patches (CNN) or even per pixel
- Bottom-up network could look at nearby location but spatial locality could also be done only by the attention mechanism
Training
- Denoising autoencoder: reconstructing corrupted image (missing certain regions)
- To encourage islands of new identity: regularizer based on contrast learning
  - Crops from same image should agree, from different images disagree -> needs to be done on scene level not lower
Represent coordinate transformations: it’s not necessary to have explicit part-whole coordinate transformations
- Better have it implicit in higher dimensional embedding than explicit in low level (can’t model uncertainty)
For video: don’t need to converge for each frame, can move in time within video during convergence
- I.e. Few iteration steps per each video frame, if changes not too rapid -> should still reach stable higher levels

Deep learning notes 05: Unsupervised vision with DINO

Tue, 11 May 2021 19:50:44 +0000

DINO: Emerging Properties in Self-Supervised Vision Transformers

Unsupervised learning regime for vision (self attention, 8x8 patches) transformers
- Intermediate representation clusters pictures of similar labels together (without seeing labels)
- Capable of object detection and masking (attention mask segments objects very well)
- Capable of classification (output KNN to known labeled examples)
- Copy detection, image retrieval, … -> good similarity measure
Attention masks for CLS token: the token that contains final representation (doesn’t have image patch on input, to not bias)
Self-supervised learning: self-distillation without labels
Negative samples learning:
- Take anchor patch and patch A from one image, and patch B from second image
- Give all three patches to the model, tell it which is anchor patch
- Ask whether A or B is from the same as anchor
Self-supervised without negative samples learning
- Use only one image, augment in multiple ways (BYOL) -> produce two versions for teacher and student
  - Global crops: > 50 % of the image
  - Local crops: < 50 % of the image
  - Rotations, color-jitters, …
- Pass each one version through teacher, one through student
  - Note: Actually pass both through both, loss is combination of cross-difference
- Loss is the difference between end image representations (CLS output)
  - Same image, only differently augmented -> should have similar representation
  - To mitigate collapse to single repre. -> different models for teacher and student
- Only train (backprop) student, build teacher as exponential average of students
Teacher only uses global cropping
- If student has local crop -> student learns that its patch should match the whole with more context
- -> forces the model to learn part-whole relationship & representing the whole image
Teacher maintains running average of all representations it sees -> subtracts it from its representation
- ~normalization, helps against collapse
Representation has softmax with temperature at the end
- Dimensionality of softmax is arbitrary: don’t have explicit labels (unsupervised) -> who knows how many
- Teacher has sharpening -> more peaked distribution -> forcer larger differences between diff. outputs
- Softmax is not common in unsupervised -> forces model to come up with “its own classes”
Versus supervised learning
- Supervised has way more noisy / overfitted attention mask -> hyper optimization on the task at hand
Why does it work?
- Augmentations: in computer vision they’re super important ~ that’s where the human prior is
  - What’s augmented away doesn’t matter
Dataset: there’s always an explicit object of interest -> how we take pictures brings prior

SWE notes 01: Types as weather

Tue, 11 May 2021 19:50:42 +0000

Types are like the Weather, Type Systems are like Weathermen - Matthias Felleisen

Types are language of prediction by the programmer what the program will do
Type systems check these prediction
Code is written for others to understand but also to be run on computers
All developers think types while their create code (more or less precise, but still)
- Capturing these thoughts in comments is problematic -> not checked -> become wrong
- Types are checked automatically
Only type inference added to untyped language fundamentally doesn’t work
- If things go wrong -> super-hard to have reasonable error messages
Instead add gradual typing system
- Allow adding types incrementally throughout codebase
- Idiomatic: just adding types, not changing code
- Strive for reasonable error messages
What should happen when part is types and there’s error in untyped land
- In typed racket: It tells user what happened on the typed/untyped boundary (through value proxies)
- Can also provide profiling info w.r.t to specific values
Contracts are good but they are very much not free during runtime
- Problem with higher order objects (lazy streams, first class functions, …) -> need to allocate
- Good for more complex checking hard to encode in types
Idea: JITs could exploit dependent types (just like compilers exploit static types)

Deep learning notes 04: Perceivers and Branch specialization

Wed, 14 Apr 2021 21:10:21 +0000

Perceiver: General Perception with Iterative Attention (arxiv)

CNN: Locality exploited by sliding window
ViT: Divide picture into (local) patches, attend over them
Traditional: self-attention computation and memory n^2 in the number of tokens (|K|*|Q|)
- NLP: 1000s tokens
- CV: ideally » 50k pixels
Originally NLP transformers: encoder (cross attention) & decoder (self attention)
- Encoder on input: attends only to input tokens, same number of keys, values and queries
- Decoder on output: attends to both input and output tokens, input only produces keys and values; not queries
- -> different amount of (keys, values) and (queries): computational requirements are not strictly quadratic
Perceiver: split (queries) and (keys, values) for vision classification models
- Stack of cross-attention (with input), and few self-attentions (mix, compute) repeated multiple times
- Cross-attention: queries not based on input, but with arbitrarily smaller dimension N*D N is ~1000
  - Dim of N*M: way smaller than M^2, only linearly on input -> allows video, not-patched images, sound, …
- Self-attention: queries as well as keys, values based on cross-att.’s queries dim -> arbitrarily small
  - Dim of N^2: independent on input dimensions; uses output of prev. step for values, keys
Multiple layers of this cross attentions, self-attentions stack
- Each cross-attention uses the same input image for calculating (using different weights) keys, values
- Weights for keys, values (from input), and queries can be shared across repeats -> essentially recurrent neural network
Initial queries can be random or learned; queries in later layers are calculated based on earlier layer results
Interpretation:
- Queries: what we would want to know; represent channels
- Keys: what each pixel represents/offers
Fourier features for positional encoding, not learned
Results comparable to ResNet without any picture assumptions (apart from 2d positional encoding)
- ~50 layers, number of parameters comparable to ResNet
Attention maps possibly static / dependent only on location instead of input content

Branch specialization: Similar features cluster together

When CNNs are branched into multiple sets of channels that don’t allow cross-information sharing -> specialization
- E.g. on initial portion of AlexNet (split into streams two due to GPU memory limits)
- Features are not organized randomly as within a normal layer, but grouped/clustered for each branch
- First group: Black and white Gabor filters, second group: low frequency color detectors
Possible explanation:
- First part of branch is incentivized to form features relevant to second part
- <=>
- Second part prefers features which the first half provides primitives for
Inception (multiple parallel blocks) and residual networks (unroll parallelly) also feature separate sets of channels
- Residual networks sidestep requirement to have a lot of parameters to mix values between branches
- Bigger convolution (i.e. smaller branches) tend to be more specialized (e.g. 5x5 for Inception)
- Inception happens even across multiple depths:
  - mixed3a_5x5: BW vs Color, brightness, …
  - mixed3b_5x5: curve related, boundary detectors, fur/eye/face detectors, …
  - mixed4a_5x5: complex 3D shapes/geometry
  - -> Very unlikely to happen by chance e.g. for mixed3a_5x5 ~ 1/10^8
Specialization is consistent across architectures and tasks
- The two groups on Alexnet no matter what you train it on (Places, …)
- Specialized curvature group also common across architectures / datasets
Hypothesis: Branching just surfaces structure that already exists
- Test: weights between 1st and 2nd conv layers -> SVD -> singular vectors: frequency and colors
- The largest dimension of variation in which neurons connect to which in the next layer
Idea: parallels to neuroscience and brain region specialization